BabelArena: A Large-Scale Multilingual Benchmark for LLM Agents

2026-09-20

AI Summary

为了解决多语言环境下大语言模型代理评估不足的问题，通过BabelFlow方法构建了BabelArena基准，涵盖23种语言，用于评估模型在多语言任务中的表现。

Abstract

Large language model (LLM) agents increasingly execute multi-step workflows through tool use and interaction with users and environments. However, current agent evaluations are largely English-centric, limiting our understanding of agent capabilities in multilingual settings. We introduce BabelFlow, a benchmark-general agentic workflow that adapts existing agent benchmarks to new languages by analyzing runtime dependencies, coordinating structure-preserving translation, and combining multi-layer verification with human review to preserve task and evaluation semantics. Using BabelFlow, we construct BabelArena, a task-aligned benchmark comprising 16,146 instances derived from 702 canonical tasks across four benchmark families, 13 domains, and 23 languages. Experiments with five frontier models show that no single model dominates across benchmark families and that cross-language disparities extend well beyond task success. Lower-resource languages exhibit distinct failure patterns, with larger shares of tool-use and control-flow errors rather than answer-quality errors alone, pointing to gaps in reliable task execution across the resource levels of these languages. On the same tasks, agents in low-resource languages also consume substantially more tokens than in English (up to roughly twice the input) without proportional increases in interaction length, and language consistency degrades further on tasks requiring structured output, where switches are directed overwhelmingly toward English. We believe BabelArena provides a foundation for advancing research on reliable and efficient multilingual agents.

Problem

• multilingual
• large language model
• agent evaluation
• cross-language disparities
• low-resource languages

Innovation

• BabelFlow
• Multilingual Benchmark
• Cross-Language Disparities
• Tool-Use Errors
• Control-Flow Errors