MADE: Beyond Scoring via a Multilingual Agentic Diagnosing Engine for Fine-Grained Evaluation Insights

2026-06-05

AI Summary

This work addresses the limited fine-grained diagnostic capability of existing large-scale multilingual evaluations, which hinders effective model optimization. The authors propose the first reusable multilingual agent-based diagnostic framework, decomposing post-evaluation analysis into five stages: planning, aggregation, instance inspection, cross-cultural reflection, and report generation. Integrating an expert knowledge base with multilingual understanding and culture-aware modules, the framework enables deep attribution across 33 model families, 11 benchmarks, 26 languages, and 34 cultural contexts. Leveraging an expert-driven diagnostic set comprising 54 queries across 15 languages, the approach translates scores into actionable guidance, yielding diagnostic reports that outperform the strongest baseline by 47% in quality and prevail in 87.9% of pairwise comparisons against human experts. The study further distills four key insights regarding deployment strategies, iterative refinement, and cross-cultural risk mitigation.

Abstract

Multilingual and multicultural benchmarks now cover dozens of languages and model families, but the resulting score landscapes remain metric-rich and insight-poor, necessitating fine-grained multilingual post-evaluation diagnosis. However, single LLMs and open-ended agents are easily swamped by the long, noisy diagnostic input, and no reusable taxonomy exists for it. To address this, we propose MADE, a Multilingual Agentic Diagnosing Engine that decomposes post-evaluation analysis into planning, aggregate analysis, instance-level case inspection, multilingual and cultural reflection, and grounded report synthesis. MADE is paired with an expert-led 54-query and 15-language diagnostic set, evaluated on top of a large-scale multilingual evaluation substrate (33 model families, 11 benchmarks, 26 languages, 34 cultures, 8.66M evaluation records). Experiments show that MADE outperforms the strongest shared baseline by 47% in diagnosis report quality and is preferred by human multilingual experts in 87.9% of pairwise comparisons. Applied with multilingual experts, MADE further surfaces four actionable findings on deployment, iteration, and cross-cultural pitfalls, turning benchmark score tables into model-selection and remediation guidance.

Problem

• multilingual evaluation
• fine-grained diagnosis
• post-evaluation analysis
• benchmark insights
• cross-cultural assessment

Innovation

• Multilingual Diagnosis
• Agentic Evaluation
• Fine-Grained Analysis
• Cross-Cultural Reflection
• Diagnostic Engine