D'ej`a Vu: Multilingual LLM Evaluation through the Lens of Machine Translation Evaluation

2025-04-16

AI Summary

解决多语言大模型生成能力评估缺乏全面性和科学性的问题，通过借鉴机器翻译评估方法，提出改进建议和评估标准。

Technology Category

• Natural Language Processing: Machine Translation, Multilinguality, Cross-Lingual NLP
• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Generation capabilities and language coverage of multilingual large language models (mLLMs) are advancing rapidly. However, evaluation practices for generative abilities of mLLMs are still lacking comprehensiveness, scientific rigor, and consistent adoption across research labs, which undermines their potential to meaningfully guide mLLM development. We draw parallels with machine translation (MT) evaluation, a field that faced similar challenges and has, over decades, developed transparent reporting standards and reliable evaluations for multilingual generative models. Through targeted experiments across key stages of the generative evaluation pipeline, we demonstrate how best practices from MT evaluation can deepen the understanding of quality differences between models. Additionally, we identify essential components for robust meta-evaluation of mLLMs, ensuring the evaluation methods themselves are rigorously assessed. We distill these insights into a checklist of actionable recommendations for mLLM research and development.

Problem

• Evaluating multilingual LLMs lacks comprehensive and rigorous standards
• Applying machine translation evaluation practices to improve mLLM assessment
• Developing robust meta-evaluation components for reliable mLLM quality analysis

Innovation

• Adopting MT evaluation standards for mLLMs
• Enhancing multilingual generative model assessments
• Developing checklist for robust mLLM meta-evaluation