GlotEval: A Test Suite for Massively Multilingual Evaluation of Large Language Models

2025-04-05

AI Summary

Current LLM evaluations exhibit strong English and high-resource language bias, leading to distorted assessments of multilingual—particularly low-resource language—capabilities. To address this, we propose GlotEval, a lightweight multilingual evaluation framework covering dozens to hundreds of languages across seven core NLP tasks. GlotEval introduces two key innovations: (1) language-customized prompt templates and (2) a non-English-centric machine translation evaluation mechanism, enabling English-decoupled, fine-grained capability diagnostics. Leveraging language-aware prompt engineering, cross-lingual consistency calibration, and low-resource data augmentation, GlotEval conducts systematic evaluations across数十 low-resource languages. Experimental results demonstrate significantly improved accuracy in identifying multilingual performance bottlenecks. By mitigating English-centric biases and enhancing reproducibility, GlotEval supports fairer, more robust, and globally inclusive LLM assessment practices.

Technology Category

• Natural Language Processing: Machine Translation, Multilinguality, Cross-Lingual NLP
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Evaluation and Analysis

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Large language models (LLMs) are advancing at an unprecedented pace globally, with regions increasingly adopting these models for applications in their primary language. Evaluation of these models in diverse linguistic environments, especially in low-resource languages, has become a major challenge for academia and industry. Existing evaluation frameworks are disproportionately focused on English and a handful of high-resource languages, thereby overlooking the realistic performance of LLMs in multilingual and lower-resource scenarios. To address this gap, we introduce GlotEval, a lightweight framework designed for massively multilingual evaluation. Supporting seven key tasks (machine translation, text classification, summarization, open-ended generation, reading comprehension, sequence labeling, and intrinsic evaluation), spanning over dozens to hundreds of languages, GlotEval highlights consistent multilingual benchmarking, language-specific prompt templates, and non-English-centric machine translation. This enables a precise diagnosis of model strengths and weaknesses in diverse linguistic contexts. A multilingual translation case study demonstrates GlotEval's applicability for multilingual and language-specific evaluations.

Problem

• Evaluating large language models in diverse linguistic environments
• Addressing bias towards English and high-resource languages
• Providing multilingual evaluation for low-resource languages

Innovation

• Lightweight framework for multilingual evaluation
• Consistent benchmarking across diverse languages
• Language-specific prompts and non-English MT