Benchmarking Diarization Models

2025-09-30

AI Summary

Speaker diarization addresses the fundamental problem of determining “who spoke when,” yet current approaches exhibit substantial errors—particularly under multi-speaker, multilingual, and heterogeneous acoustic conditions—and these errors propagate to downstream tasks. This study systematically evaluates five state-of-the-art end-to-end models—including PyannoteAI and DiariZen—across four diverse multilingual datasets (English, Chinese, German, Japanese, Spanish), totaling 196.6 hours. We identify two primary error sources for the first time: undetected speech segments and speaker identity confusion under high speaker counts. PyannoteAI achieves the best performance with a 11.2% diarization error rate (DER); DiariZen attains the lowest DER (13.3%) among open-source models, establishing it as the most competitive开源 alternative. The work provides an interpretable bottleneck analysis and empirically grounded boundary validation to guide model optimization.

Technology Category

Machine Learning: Large Multimodal Models (LMMs)Natural Language Processing: Conversational AI/Dialog SystemsKnowledge Representation and Reasoning: Diagnosis and Abductive Reasoning

Application Category

Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web searchWeb Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web dataSemantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Speaker diarization is the task of partitioning audio into segments according to speaker identity, answering the question of "who spoke when" in multi-speaker conversation recordings. While diarization is an essential task for many downstream applications, it remains an unsolved problem. Errors in diarization propagate to downstream systems and cause wide-ranging failures. To this end, we examine exact failure modes by evaluating five state-of-the-art diarization models, across four diarization datasets spanning multiple languages and acoustic conditions. The evaluation datasets consist of 196.6 hours of multilingual audio, including English, Mandarin, German, Japanese, and Spanish. Overall, we find that PyannoteAI achieves the best performance at 11.2% DER, while DiariZen provides a competitive open-source alternative at 13.3% DER. When analyzing failure cases, we find that the primary cause of diarization errors stem from missed speech segments followed by speaker confusion, especially in high-speaker count settings.

Problem

• Evaluating speaker diarization models' performance across diverse datasets
• Identifying primary error causes like missed speech and speaker confusion
• Benchmarking multilingual audio processing in varied acoustic conditions

Innovation

• Evaluated five state-of-the-art diarization models
• Used 196.6 hours multilingual audio datasets
• Analyzed missed speech and speaker confusion errors