Probing Speaker Identity Sensitivity in Audio Deepfake Detectors

2026-07-23

AI Summary

This work addresses the significant performance degradation of audio deepfake detectors in cross-dataset scenarios, primarily attributed to their overreliance on speaker identity rather than genuine synthesis artifacts. To diagnose this issue at inference time without ground-truth labels, the authors propose Identity Sensitivity Score (ISS)—a metric that quantifies a detector’s dependence on speaker identity by measuring output variations across different speaker contexts. Leveraging voice conversion, a reference speaker pool, and score analysis, ISS establishes an interpretable framework for sensitivity assessment. Experiments demonstrate that ISS achieves an AUC of 0.954 in predicting misclassifications across two detectors and datasets; further validation via voice conversion reveals that high-ISS samples exhibit 19–30 times stronger response magnitudes, effectively flagging errors caused by identity-dependent biases.

Abstract

Audio deepfake detectors are trained to distinguish genuine speech from synthetic speech and often perform well on standard benchmarks. Yet the same detector that achieves less than 1% error on one dataset can see its error rate increase twentyfold when evaluated on a different dataset. We argue that one contributing factor is speaker-identity reliance: standard training corpora correlate speaker identity with the genuine/synthetic label, allowing detectors to partially rely on speaker-related cues rather than synthesis artifacts alone. We propose the Identity Sensitivity Score (ISS), a per-utterance diagnostic that quantifies how much a detector's output changes across different speaker identity contexts. ISS requires no ground-truth labels at inference time and can be computed from the detector score and a pool of reference speaker examples. Across two detectors and two datasets, incorrectly classified utterances have ISS scores 29 to 52 times higher than correctly classified utterances, and ISS alone predicts misclassification with area-under-curve (AUC) up to 0.954. To test whether ISS actually captures identity-sensitive behavior rather than serving only as a proxy for prediction confidence, we apply voice conversion to 500 utterances and measure the resulting detector-score shift. Utterances flagged as identity-sensitive by ISS respond 19 to 30 times more strongly to this manipulation than utterances flagged as stable. These results position ISS as a practical inference-time diagnostic for speaker-dependent failure analysis in audio deepfake detection.

Problem

• audio deepfake detection
• speaker identity
• generalization
• dataset shift
• synthetic speech

Innovation

• Identity Sensitivity Score
• audio deepfake detection
• speaker identity reliance
• voice conversion
• inference-time diagnostic