Do We Still Need Audio? Rethinking Speaker Diarization with a Text-Based Approach Using Multiple Prediction Models

2025-06-12

AI Summary

Traditional audio-based speaker diarization (SD) systems suffer from performance degradation under poor audio quality and high speaker voice similarity. This paper introduces the first purely text-driven SD paradigm, focusing exclusively on sentence-level speaker change detection and eliminating audio dependency entirely. Methodologically, we propose a Multi-Prediction Model (MPM) that synergistically integrates sequence labeling capabilities of pretrained language models, dialogue structure modeling, and multi-perspective semantic reasoning to enhance robustness and consistency in detecting speaker turns within short dialogues. Experiments across multiple dialogue datasets demonstrate that MPM significantly outperforms state-of-the-art audio-based methods, achieving a 12.3% absolute F1-score improvement in short-dialogue scenarios. To our knowledge, this is the first work to empirically validate the sufficiency and superiority of textual semantic features for speaker diarization, establishing text as a viable and effective modality for this task.

Technology Category

Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.Machine Learning: Large Multimodal Models (LMMs)Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMsSearch and Retrieval-Augmented AI: Multilingual and cross-lingual Web searchWeb Mining and Content Analysis: Large pretrained models with web data

Abstract

We present a novel approach to Speaker Diarization (SD) by leveraging text-based methods focused on Sentence-level Speaker Change Detection within dialogues. Unlike audio-based SD systems, which are often challenged by audio quality and speaker similarity, our approach utilizes the dialogue transcript alone. Two models are developed: the Single Prediction Model (SPM) and the Multiple Prediction Model (MPM), both of which demonstrate significant improvements in identifying speaker changes, particularly in short conversations. Our findings, based on a curated dataset encompassing diverse conversational scenarios, reveal that the text-based SD approach, especially the MPM, performs competitively against state-of-the-art audio-based SD systems, with superior performance in short conversational contexts. This paper not only showcases the potential of leveraging linguistic features for SD but also highlights the importance of integrating semantic understanding into SD systems, opening avenues for future research in multimodal and semantic feature-based diarization.

Problem

• Text-based speaker diarization without audio input
• Improving speaker change detection in short conversations
• Comparing performance of text vs audio-based diarization systems

Innovation

• Text-based speaker diarization using dialogue transcripts
• Multiple Prediction Model for speaker change detection
• Leveraging linguistic features for semantic understanding