IndoBias: A Dual Track Culturally Grounded Benchmark for LLMs Bias Evaluation in Indonesian Languages

2026-05-31

AI Summary

This study addresses the lack of systematic evaluation of bias in large language models within Indonesia’s multilingual and multicultural context. It introduces the first benchmark for culturally embedded bias assessment, encompassing Indonesian as well as regional languages—Javanese, Sundanese, and Makassarese—by integrating sociological frameworks (SPI, O*NET, WGI) with contrastive pair construction and generative probing techniques. Bias is quantified and attributed through human annotation and large-scale corpus analysis. Findings reveal that decoder-only models exhibit a preference for prototypical expressions in Indonesian, while religious and ideological biases are more pronounced in regional languages. Moreover, uncurated corpora such as Common Crawl introduce substantially more bias than Wikipedia, highlighting the critical role of pretraining data sources and local language inclusion in shaping model bias.

Abstract

Despite being home to more than 1300 ethnic groups and 700 indigenous languages, bias in Large Language Models has not been fully studied in Indonesia, thus leaving a critical gap in evaluating representational fairness and localized stereotypes within its uniquely vast, multilingual, and diverse sociocultural landscape. To address this, we introduce IndoBias as a culturally-grounded bias benchmark to assess LLMs bias in Indonesian and three local languages: Javanese, Sundanese, and Makasar. IndoBias features dual perspective evaluation tracks: depth-oriented (with contrastive-pairs) and breadth-oriented (with generation-based), where the latter is grounded in social science frameworks (SPI, O*NET, and WGI). Our results show that existing LLMs -- particularly decoder models -- exhibit strong bias towards prototypical sentences in Indonesian, while local languages suffer higher bias under Ideology and Religion category. We also find that LLMs responses exhibit a non-uniform Stereotype Polarity when prompted with various local entities. Finally, we discover that, in Indonesian, Common Crawl texts introduce more bias during pretraining, compared to human-reviewed article texts (e.g., Wikipedia, News), whereas introducing local languages to pretraining generally increases bias. This work highlights the importance of studying bias in culture-specific context. Warning: This paper contains example data that may be offensive, harmful, or biased.

Problem

• bias
• Large Language Models
• Indonesian languages
• cultural context
• stereotypes

Innovation

• culturally-grounded benchmark
• dual-track evaluation
• multilingual bias
• local language representation
• stereotype polarity