Found in Translation: Measuring Multilingual LLM Consistency as Simple as Translate then Evaluate

2025-05-28

AI Summary

Multilingual evaluation of large language models (LLMs) is hindered by reliance on costly human annotation and the difficulty of quantifying open-ended generation tasks. Method: This paper proposes a “translation-then-evaluation” framework: identical queries are translated across languages and fed into the LLM; semantic similarity and empathy scoring models then quantify cross-lingual consistency along two decoupled dimensions—information preservation and empathetic expression. Contribution/Results: The framework enables the first annotation-free, disentangled, and quantifiable assessment of multilingual consistency, supporting 30 languages spanning major language families and writing systems, with task-agnostic applicability and low computational cost. Experiments reveal significant consistency degradation in Slavic languages, Arabic-script languages, and low-resource languages; for certain language pairs, information preservation errors exceed 40%, exposing fundamental limitations in current LLMs’ multilingual capabilities.

Technology Category

• Natural Language Processing: Machine Translation, Multilinguality, Cross-Lingual NLP
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Language and Vision

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Large language models (LLMs) provide detailed and impressive responses to queries in English. However, are they really consistent at responding to the same query in other languages? The popular way of evaluating for multilingual performance of LLMs requires expensive-to-collect annotated datasets. Further, evaluating for tasks like open-ended generation, where multiple correct answers may exist, is nontrivial. Instead, we propose to evaluate the predictability of model response across different languages. In this work, we propose a framework to evaluate LLM's cross-lingual consistency based on a simple Translate then Evaluate strategy. We instantiate this evaluation framework along two dimensions of consistency: information and empathy. Our results reveal pronounced inconsistencies in popular LLM responses across thirty languages, with severe performance deficits in certain language families and scripts, underscoring critical weaknesses in their multilingual capabilities. These findings necessitate cross-lingual evaluations that are consistent along multiple dimensions. We invite practitioners to use our framework for future multilingual LLM benchmarking.

Problem

• Evaluating multilingual LLM consistency without expensive annotated datasets
• Assessing cross-lingual response predictability via Translate-then-Evaluate
• Identifying inconsistencies in LLM performance across 30 languages

Innovation

• Translate then Evaluate strategy
• Cross-lingual consistency framework
• Multilingual LLM benchmarking tool