Training-Free Affinity Fusion of Neural and Embedding-Based Speaker Diarization

2026-09-30

AI Summary

This study addresses the incompatibility of intermediate representations and the reliance on additional training when fusing neural and embedding-based speaker diarization systems. To overcome these limitations, this work proposes a training-free affinity fusion method. By conditioning local representations through neural partitioning, the approach constructs a continuous affinity matrix that is subsequently merged with acoustic affinities to perform global clustering. This enables the seamless integration of heterogeneous systems without requiring extra training, shared embedding spaces, or label alignment. Experimental evaluations on the AMI and CALLHOME datasets demonstrate that the proposed method significantly reduces the diarization error rate (DER), effectively enhancing speaker attribution in transcription tasks.

Abstract

Speaker diarization systems based on speaker embeddings and neural diarization exploit complementary forms of speaker information, but their intermediate representations are not directly compatible. We introduce Training-Free Affinity Fusion (TFAF), which integrates the speaker structure inferred by a neural diarizer into an embedding-based diarization system. The neural speaker partition is used to condition local speaker representations, from which we construct a continuous affinity matrix and combine it with the embedding-based acoustic affinity before a single global clustering step. The method requires no additional training, shared embedding space, speaker-label alignment, or hard transfer of the neural diarizer's speaker count. Experiments on AMI and CALLHOME show consistent DER improvements over both constituent systems; on AMI, fusion also improves speaker-attributed transcription. Ablations show that the neural speaker partition accounts for most of the gain, while retaining the continuous embedding-based affinities provides additional benefit over hard partition fusion.

Problem

• Speaker Diarization
• Affinity Fusion
• Neural Diarization
• Speaker Embeddings

Innovation

• Training-Free Affinity Fusion
• Speaker Diarization
• Neural Diarizer
• Affinity Matrix
• Global Clustering