Beyond Bias: Evaluating Cultural AI Through Participation and Interpretation

1Introduction

Generative AI systems increasingly participate in the production of culture, shaping how communities are represented and which traditions receive visibility. As Crawford (2021) argues, AI must be understood in relation to the material resources and power structures that underpin its development. For example, image-generation models may reproduce stereotypical depictions of Indigenous communities or misrepresent regional clothing and cultural practices, influencing how these communities are perceived by wider audiences. When such images circulate through educational resources, digital archives, or public campaigns, they can reinforce inaccurate representations and obscure the diversity of lived cultural experiences. These concerns make it important to consider whose knowledge informs AI systems, whose interpretations they privilege, and how communities can participate in shaping the cultural representations these technologies produce. Evaluation of generative AI has largely centred on harm mitigation. Existing research has addressed issues such as bias, misinformation, toxicity, fairness, and alignment with human values (Bender et al., 2021; Buolamwini & Gebru, 2018). These approaches remain essential, particularly given the structural inequalities embedded within large-scale datasets and machine learning pipelines. However, they define cultural success narrowly, as the absence of harmful output. The larger question here is how AI can be more meaningful in its cultural representation.

The limitations of existing approaches become especially visible in culturally situated contexts. Models tend to default to Western visual and linguistic conventions, collapsing local traditions, minority aesthetics, and situated knowledge into generic imagery (Birhane, 2021; Crawford, 2021). What survives is often a set of surface style cues, cut off from the practices and communities behind them.

Here we report on Beyond Bias, a joint project of Gooey.AI and Goethe-Institut India that tests participatory routes to cultural AI. Rather than treating culture as an external layer of evaluation, the project embeds cultural considerations directly within dataset creation, model training, interface design, and governance processes. Through collaborative workshops, artist-led fine-tuning, and co-authored manifestos, the initiative examines how AI systems might support culturally grounded forms of expression while preserving community agency and interpretive context.

A related paper by the same authors examines Beyond Bias through participatory workshops, reflective AI tools, and cultural practice (Prasad et al., 2026). The two papers draw on the same initiative, workshop data and figures, and share some descriptive background text and framework dimensions. The present paper contributes the evaluation framing and does not report the reflective-practice findings of that paper.

The paper makes three contributions:

• A brief account of the Beyond Bias initiative, whose participatory methodology and workshop findings are reported in full in Prasad et al. (2026).
• An interpretive framing of AI systems that positions generative models as technologies involved in meaning-making rather than neutral systems of pattern reproduction (Hall, 1997).
• A framework for evaluating cultural AI systems based on cultural integrity, participation and agency, and interpretive capacity.

Ultimately, this work suggests that evaluating cultural AI requires looking beyond frameworks focused only on reducing harm, toward approaches that also consider cultural context, stewardship, interpretation, and what meaningful cultural outcomes might look like.

1.1 From Bias Mitigation to Cultural AI

Studies of generative AI systems have highlighted disparities across race, gender, geography, and language. For instance, image models have been shown to perform less accurately on darker skin tones, and they often default to Western cultural norms and English-language contexts. These concerns have led to the development of mitigation strategies such as dataset balancing, filtering systems, fairness constraints, and model alignment techniques. While these approaches address important harms, they are often focused more on avoiding mistakes than on supporting meaningful cultural participation. In many cases, “cultural inclusion” is treated simply as having diverse representation in datasets, without considering how culture is interpreted, understood in context, or shaped by the communities involved.

From this perspective, generative models do more than just repeat patterns from training data. They actively shape culture by creating images, stories, language, and visual styles that influence how people understand the world. For example, AI image generators can shape popular ideas of beauty, fashion, religion, or regional identity through the kinds of images they produce and circulate online. Their outputs resemble cultural artefacts traditionally studied within the humanities, requiring interpretive rather than purely statistical forms of evaluation (Hall, 1997).

This shift raises important questions about how AI systems should be evaluated. If AI increasingly helps interpret and shape culture, then evaluation cannot depend only on numerical benchmarks or accuracy scores. Cultural meaning depends on context, history, and lived experience. For example, an AI-generated image of a religious ritual may appear visually accurate but still misrepresent its cultural meaning or significance. Evaluating cultural performance therefore requires attention to interpretation, aesthetics, provenance, and community experience.

Participatory and human-centered approaches to AI development offer one possible response to these limitations. Participatory design traditions emphasize co-creation, contextual expertise, and stakeholder involvement within technical systems (Sanders & Stappers, 2008). However, such methods have often remained peripheral within mainstream generative AI development, where datasets are typically extracted at scale with limited transparency or community governance (Couldry & Mejias, 2019). Beyond Bias builds upon participatory design approaches by integrating collaborative cultural practices directly into dataset curation, model fine-tuning, and governance structures for generative AI systems.

1.2 Beyond Bias: A Participatory Framework for Cultural AI

Beyond Bias, begun in March 2025, is a partnership between Gooey.AI and Goethe-Institut India that tests participatory approaches to generative AI in different cultural settings. The project brought together artists, researchers, designers, and technologists from India, Germany, the United Kingdom, and the United States through workshops, roundtables, and collaborative experimentation. It featured over 200+ workshop participants and 24 member organisations in the stakeholders consortium.

A central objective of the initiative was to examine how communities might participate more directly in shaping the datasets, tools, and governance structures underlying generative AI systems. The initiative was structured around three interconnected components:

• Manifesto and Collective Governance: The initiative first convened a consortium of institutional stakeholders from the US, UK, EU and India. Two roundtables produced an open manifesto setting out principles for culturally grounded AI. The manifesto outlined principles addressing culturally grounded representation, transparent AI development, ecological responsibility, equitable compensation, and community stewardship. The manifesto set out ethical commitments and shaped the tools built for cultural practitioners. It was also a trial of shared governance, in which authorship and decisions were held collectively.
• Participatory Dataset Creation: One core component of the project involved the collaborative creation of culturally situated datasets. Participants contributed image collections reflecting specific artistic traditions, visual languages, and regional aesthetics frequently underrepresented within dominant generative AI models. Datasets were treated as curated, interpretive work, so choices about inclusion, categorisation and context became design decisions. Decisions about what to include, how to categorize materials, and how to provide context became an important part of the cultural design process. Participants created innovative images and videos that featured variations on Mughal miniature styles, Japanese print styles, German folk art and Lascaux cave paintings, or among unique aesthetic styles from their own practices. Datasets, ethically sourced, were used to fine-tune generative image models and train LoRAs on these styles. Across nine workshops, participants ran 1,640 generations that captured specific cultural aesthetics and visual traditions more closely.

• Accessible Tooling and Model Fine-Tuning: The initiative also developed an AI Image Trainer tool and Video Generation Tool that enabled participants without extensive technical expertise to fine-tune generative models using curated datasets. To support under-resourced and marginalised creators working in different languages, we also developed a prompting tool that translated prompts from local languages like Hindi into English so they could be used with our existing visual tools. The tool lowered barriers to experimentation by simplifying model training workflows while also providing transparency regarding computational and environmental costs.
• This approach redistributed aspects of technical agency toward artists and cultural practitioners typically excluded from model development processes and made them co-creators. Participants could actively shape how models interpreted visual forms rather than functioning solely as end-users of closed systems. Importantly, the project treated fine-tuning not just as a technical process, but as a way of negotiating between datasets, AI models, interfaces, and cultural context. Participants felt that the fine-tuned outputs reflected cultural details and visual aesthetics more accurately than the outputs generated by standard AI models by big tech.

Together, these practices positioned Beyond Bias as both a technical and cultural intervention into how generative AI systems are designed, evaluated, and governed.

2Evaluating Cultural AI

Existing evaluation frameworks for generative AI systems primarily focus on accuracy, alignment, safety, and fairness. While these metrics remain necessary, they are insufficient for assessing systems that increasingly participate in cultural production. A model may avoid explicit bias while still flattening cultural specificity, reproducing dominant aesthetics, or excluding communities from participation in system design. Similarly, representational diversity alone does not guarantee contextual understanding or meaningful cultural engagement. Based on insights from Beyond Bias, we propose three dimensions for evaluating cultural AI systems:

• Cultural Integrity: Cultural AI systems should preserve and support the specificity of cultural forms rather than collapsing them into generalized or aestheticized representations. This includes attention to provenance, contextual meaning, historical relationships, and community knowledge. Cultural integrity requires more than simply adding visual styles or symbols from a culture into AI systems. The systems also need to understand the context and meaning behind them. For example, an AI model may be able to generate clothing that looks like a traditional sari, but still fail to capture its regional history, social meaning, or the cultural practices connected to how it is worn.
• Participation and Agency: Communities should have meaningful influence over how datasets are created, how models are trained, and how outputs are deployed. Participation must extend beyond representation within datasets to include AI literacy, governance, authorship, and decision-making power. This dimension emphasizes AI development as a collaborative process rather than a purely extractive one.
• Interpretive Capacity: Generative AI systems increasingly shape how cultural meaning is interpreted, rather than simply repeating patterns from data. Evaluation should therefore consider whether these systems can support context-aware interpretations informed by curated datasets, human input, and cultural expertise. This approach recognises that cultural meaning is shaped through relationships, history, and context, and cannot be fully measured through statistical performance alone.

Together, these dimensions propose a broader evaluative framework for cultural AI that incorporates qualitative, humanistic, and participatory criteria alongside technical performance.

2.1 Tensions and Limitations

While participatory approaches to cultural AI offer new possibilities for representation and agency, they also introduce significant tensions. Questions of authorship and ownership remain unresolved when models are trained on collaboratively curated cultural datasets. The generation of synthetic outputs derived from community knowledge raises concerns regarding consent, attribution, and intellectual property (Benjamin, 1935/2008). Participants frequently noted that mainstream image-generation systems reproduced Indian aesthetics through generalized visual stereotypes rather than regionally specific motifs and material traditions. Similarly, systems designed to preserve cultural forms may also enable new forms of extraction, replication, and appropriation (Couldry & Mejias, 2019). Cultural participation within AI development does not automatically eliminate structural inequalities surrounding access, labor, or institutional power.

The project also raises larger questions about how generative AI may change creative industries and cultural work. While easier fine-tuning tools can help more artists and communities participate, they may also change how creative work is produced, shared, and valued. For example, independent artists may gain new ways to create culturally specific designs, but companies could also use AI-generated versions of these styles without properly crediting or compensating the original communities. These challenges show that participatory cultural AI cannot be addressed through technical solutions alone. Ongoing governance and community oversight are also needed. As a result, evaluation frameworks should look not only at representation and model performance, but also at questions of power, labor, ownership, and accountability.

3Conclusion

As generative AI systems become increasingly embedded within cultural production, evaluating these systems requires moving beyond frameworks centered exclusively on harm mitigation. This paper presented Beyond Bias as a participatory and interpretive approach to cultural AI that integrates community engagement, collaborative dataset creation, accessible tools, and collective governance into AI system design. Rather than adding culture as an afterthought, the project integrates cultural values into the design, development, and organizational processes from the beginning.

We proposed three dimensions for evaluating cultural AI systems — cultural integrity, participation and agency, and interpretive capacity — as a framework for assessing positive cultural outcomes alongside questions of fairness and safety.

Cultural AI that is also inclusive requires attention to interpretation, cultural context, stewardship, and how communities create and share meaning. For example, an AI system may generate visually accurate traditional clothing, but still fail to represent the cultural history or social meaning connected to it. Future work should continue building participatory methods and governance structures that involve communities more directly in how AI systems are designed, trained, and evaluated. This can help create AI systems that are more context-aware, responsible, and accountable to the people and cultures they represent.