PolySpeech-100: A Large-Scale Benchmark for Speech Understanding Across 100+ Languages and Dialects

2026-05-31

AI Summary

Current evaluations of large speech models are largely confined to high-resource languages and basic transcription tasks, overlooking semantic comprehension and dialectal diversity. This work introduces a large-scale speech understanding benchmark spanning 110 languages and dialects—including 19 Chinese dialects and over 80 low-resource languages—leveraging both human-recorded and instruction-driven synthetic speech to systematically assess models’ native-level comprehension in realistic multilingual settings. Evaluations of 22 prominent models reveal, for the first time, that end-to-end architectures outperform cascaded systems in dialect understanding, while chain-of-thought prompting can be detrimental in zero-shot speech comprehension. Although open-source end-to-end models exhibit superior performance on dialects, they still lag significantly on low-resource languages. This study establishes a new standard for developing inclusive next-generation speech foundation models.

Abstract

While End-to-End (E2E) Speech-Large Language Models (Speech-LLMs) are rapidly evolving, their evaluation methodologies remain limited to the era of simple transcription. Existing benchmarks suffer from three critical limitations: a pronounced bias towards high-resource languages, a focus on low-level recognition (ASR) rather than semantic reasoning, and a neglect of regional dialects. To bridge this gap, we introduce PolySpeech-100, a massive-scale benchmark designed to assess `native-level' speech comprehension across 110 linguistic variants. We employ a novel hybrid construction pipeline that augments gold-standard human recordings with instruction-driven synthetic speech, allowing us to cover 19 distinct Chinese dialects and over 80 low-resource languages. Extensive evaluation of 22 state-of-the-art models (including Gemini-3, GPT-Audio, and Qwen2.5-Omni) yields pivotal insights. First, we demonstrate that open-source E2E models outperform Cascade (ASR+LLM) systems on heavy dialects, proving that direct audio processing preserves critical paralinguistic cues and prosodic features (e.g., intonation, stress) that are often lost in standard transcription. Second, we reveal a significant performance gap: while commercial models maintain robustness, open-source models suffer catastrophic degradation on low-resource languages. Finally, counter-intuitively, we observe that under standard zero-shot settings, Chain-of-Thought prompting frequently degrades speech understanding performance for most evaluated models, revealing a potential modality alignment gap in current architectures. PolySpeech-100 establishes a rigorous standard for the next generation of inclusive, omni-capable Speech-LLMs. The data, demo, and code are publicly available at https://github.com/YoungSeng/PolySpeech-100.

Problem

• speech understanding
• low-resource languages
• dialects
• speech-LLMs
• evaluation benchmark

Innovation

• Speech-LLM
• multilingual benchmark
• dialect understanding
• end-to-end speech modeling
• modality alignment