The Masked Advantage: Uncovering Local-Language Access to Cultural Knowledge in LLMs

2026-06-05

AI Summary

This study addresses the challenge of disentangling whether large language models’ difficulties in answering culturally grounded questions stem from limited linguistic proficiency or insufficient access to local cultural knowledge. To this end, the authors propose a controlled evaluation framework that leverages authentic cultural questions through a cross-design of question type and query language, integrated with a 1PL item response theory model. This approach enables, for the first time, the decoupling of language fluency from accessibility to local cultural knowledge across multilingual large language models. Experiments spanning 13 regions and approximately 80 models demonstrate that, after controlling for language ability, querying in the local language significantly enhances cultural knowledge retrieval—a benefit particularly pronounced in both state-of-the-art and regionally adapted models.

Abstract

Large language models are increasingly used to answer culturally grounded questions across languages, yet it remains unclear whether local cultural knowledge is better accessed through English or the local language. Existing evaluations face two key limitations: many rely on parallel template-based questions that may not reflect how cultural knowledge naturally appears, and raw accuracy conflates general language proficiency with language-conditioned knowledge access. We address these issues with a controlled framework built on real-world cultural questions collected from regional benchmarks and local sources. By crossing question type (culture-agnostic vs. culture-specific) with query language (English vs. local language), and estimating ability with a shared 1PL item response theory model, we separate proficiency from localized knowledge access. Across 13 locales and roughly 80 models, we find a consistent English advantage on culture-agnostic questions, indicating stronger English proficiency. However, after accounting for this proficiency gap, local languages show a positive knowledge-access advantage in nearly all locale-model settings. This advantage is often masked in raw accuracy but becomes more visible for frontier, regionally aligned, or language-adapted models. Our results suggest that weaker local-language performance does not necessarily imply weaker cultural knowledge; rather, local cultural knowledge may be more accessible through the local language but hidden by limited language proficiency.

Problem

• cultural knowledge
• local language
• language proficiency
• large language models
• knowledge access

Innovation

• local-language advantage
• cultural knowledge access
• item response theory
• language proficiency disentanglement
• cross-lingual evaluation