Language Equality has a Price: A Systematic Investigation of Multi-turn LLM Performance for EU-24+

2026-08-02

AI Summary

This study evaluates the capacity of large language models to achieve linguistic equity across 30 languages, including all 24 official languages of the European Union, with a focus on multi-turn dialogue performance and disparities in inference costs. The authors propose a language-agnostic evaluation framework that integrates self-play goal-oriented dialogues, programmatic automated scoring, and prompt localization techniques, enabling extension to new languages using only a fixed prompt and vocabulary. Experimental results demonstrate that proprietary models consistently outperform open-source counterparts across all languages, maintaining superiority even in low-resource settings. Non-English languages exhibit, on average, 31% higher inference costs and 10% lower scores. Although models show advantages in their native linguistic regions, this does not suffice to mitigate systemic inequities. The findings reveal that linguistic equity entails not merely broad language coverage but also addressing systematic imbalances in performance and computational cost.

Abstract

We evaluate large language models (LLMs) as language agents playing goal-directed dialogue games in self-play across 30 languages: the 24 official EU languages plus six others. Unlike static or preference-based evaluation, this paradigm is multi-turn, reference-free and programmatically scored, and because the game mechanics are language-agnostic it extends to a new language by localising a fixed set of prompt and word-list files. Evaluating nine open-weight and commercial LLMs, we find that no open-weight model covers the EU-24 well: in every official language both commercial systems outscore every open-weight model, and the two weakest average below 40 points across the EU-24. The commercial systems stay ahead even in languages with four orders of magnitude less public web text, showing that linguistic parity is achievable, but not from public crawls alone. A model's home region lifts it without closing the gap: Chinese is the strongest of all 30 languages for two Chinese-developed models, yet the best Chinese score of any model belongs to a US commercial system. Coverage is also not parity of service. Pooled over models and languages, the median non-English language costs 31% more to run than English, and scores 10% lower.

Problem

• language equality
• multilingual LLMs
• performance disparity
• cost inequality
• EU-24 languages

Innovation

• multi-turn evaluation
• language equality
• dialogue games
• cross-lingual benchmarking
• LLM cost-performance tradeoff