The Case for"Thick Evaluations"of Cultural Representation in AI

2025-03-24

AI Summary

Current evaluations of cultural representation in AI image models rely on reductive, universalist criteria, neglecting the contextual, interpretive, and community-centered nature of cultural meaning. Method: This paper introduces a “thick evaluation” framework grounded in participatory action research, co-developed through South Asian cross-cultural workshops with local communities to collaboratively define representational meaning and negotiate context-sensitive evaluation metrics. Contribution/Results: The framework shifts evaluation away from technocentric paradigms toward situated, co-constructed, and contextually attuned practices, affirming community hermeneutic authority and semantic sovereignty. It yields a transferable operational framework, community-co-designed evaluation dimensions, and practical implementation guidelines. By centering cultural subjectivity and lived experience, this work advances equitable AI assessment and offers a paradigmatic innovation for studying algorithmic cultural representation.

Technology Category

• Philosophy and Ethics of AI: Bias, Fairness & Equity
• Humans and AI: Learning Human Values and Preferences
• Knowledge Representation and Reasoning: Common-Sense Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Fairness and ethical considerations in crowd work and in human-in-the-loop AI systems
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Generative AI image models have been increasingly evaluated for their (in)ability to represent non-Western cultures. We argue that these evaluations operate through reductive ideals of representation, abstracted from how people define their own representation and neglecting the inherently interpretive and contextual nature of cultural representation. In contrast to these 'thin' evaluations, we introduce the idea of 'thick evaluations': a more granular, situated, and discursive measurement framework for evaluating representations of social worlds in AI images, steeped in communities' own understandings of representation. We develop this evaluation framework through workshops in South Asia, by studying the 'thick' ways in which people interpret and assign meaning to images of their own cultures. We introduce practices for thicker evaluations of representation that expand the understanding of representation underpinning AI evaluations and by co-constructing metrics with communities, bringing measurement in line with the experiences of communities on the ground.

Problem

• Evaluating cultural representation in AI image models
• Addressing reductive ideals in current evaluation methods
• Developing community-based thick evaluation frameworks

Innovation

• Introduces 'thick evaluations' framework
• Co-constructs metrics with communities
• Studies cultural interpretations via workshops