A Principled Framework for Evaluating on Typologically Diverse Languages

2024-07-06

arXiv.org

AI Summary

To address insufficient typological diversity and subjective sampling in multilingual NLP model evaluation, this paper proposes the first quantifiable and reproducible framework for optimizing typological diversity. Methodologically, it integrates linguistic typological features from the World Atlas of Language Structures (WALS), information-theoretic measures, and submodular optimization to enable constrained optimal language subset selection. Its core contribution lies in formulating language diversity as an explicit, differentiable objective—overcoming the inconsistency inherent in heuristic sampling strategies. Experiments demonstrate that the selected language subsets significantly outperform mainstream baselines (e.g., geographic or language-family-balanced sampling) across multiple diversity metrics and yield more robust predictions of cross-lingual generalization performance. This work provides both theoretical grounding and a practical toolkit for principled multilingual evaluation.

Technology Category

• Natural Language Processing: Machine Translation, Multilinguality, Cross-Lingual NLP
• Search and Optimization: Sampling/Simulation-based Search
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Beyond individual languages, multilingual natural language processing (NLP) research increasingly aims to develop models that perform well across languages generally. However, evaluating these systems on all the world's languages is practically infeasible. To attain generalizability, representative language sampling is essential. Previous work argues that generalizable multilingual evaluation sets should contain languages with diverse typological properties. However, 'typologically diverse' language samples have been found to vary considerably in this regard, and popular sampling methods are flawed and inconsistent. We present a language sampling framework for selecting highly typologically diverse languages given a sampling frame, informed by language typology. We compare sampling methods with a range of metrics and find that our systematic methods consistently retrieve more typologically diverse language selections than previous methods in NLP. Moreover, we provide evidence that this affects generalizability in multilingual model evaluation, emphasizing the importance of diverse language sampling in NLP evaluation.

Problem

• Develop models performing well across diverse languages.
• Evaluate multilingual NLP systems on typologically diverse languages.
• Improve generalizability through systematic language sampling methods.

Innovation

• Systematic framework for typologically diverse language sampling
• Comparison of sampling methods using diverse metrics
• Enhanced generalizability in multilingual model evaluation