VocalBridge: Latent Diffusion-Bridge Purification for Defeating Perturbation-Based Voiceprint Defenses
🤖 AI Summary
This work addresses the vulnerability of existing perturbation-based voice privacy protection methods under adaptive purification attacks, which struggle to defend against voice cloning and speaker verification threats. The authors propose a diffusion-based bridging purification framework that learns a mapping from perturbed to clean speech in the EnCodec latent space, leveraging a time-conditioned 1D U-Net architecture and cosine noise scheduling to enable efficient, text-free speech restoration while preserving speaker-discriminative characteristics. A novel lightweight temporal prior is introduced through phoneme alignment guided by the Whisper speech encoder, enhancing purification performance without requiring ground-truth transcriptions. Experimental results demonstrate that the method significantly outperforms current approaches in recovering cloneable speech, thereby exposing critical weaknesses in prevailing perturbation-based defenses and offering new directions—and challenges—for voice privacy preservation.
Application Category
📝 Abstract
The rapid advancement of speech synthesis technologies, including text-to-speech (TTS) and voice conversion (VC), has intensified security and privacy concerns related to voice cloning. Recent defenses attempt to prevent unauthorized cloning by embedding protective perturbations into speech to obscure speaker identity while maintaining intelligibility. However, adversaries can apply advanced purification techniques to remove these perturbations, recover authentic acoustic characteristics, and regenerate cloneable voices. Despite the growing realism of such attacks, the robustness of existing defenses under adaptive purification remains insufficiently studied. Most existing purification methods are designed to counter adversarial noise in automatic speech recognition (ASR) systems rather than speaker verification or voice cloning pipelines. As a result, they fail to suppress the fine-grained acoustic cues that define speaker identity and are often ineffective against speaker verification attacks (SVA). To address these limitations, we propose Diffusion-Bridge (VocalBridge), a purification framework that learns a latent mapping from perturbed to clean speech in the EnCodec latent space. Using a time-conditioned 1D U-Net with a cosine noise schedule, the model enables efficient, transcript-free purification while preserving speaker-discriminative structure. We further introduce a Whisper-guided phoneme variant that incorporates lightweight temporal guidance without requiring ground-truth transcripts. Experimental results show that our approach consistently outperforms existing purification methods in recovering cloneable voices from protected speech. Our findings demonstrate the fragility of current perturbation-based defenses and highlight the need for more robust protection mechanisms against evolving voice-cloning and speaker verification threats.
Problem
Research questions and friction points this paper is trying to address.
voiceprint defense
perturbation-based protection
voice cloning
speaker verification
adversarial purification
Innovation
Methods, ideas, or system contributions that make the work stand out.
Latent Diffusion
Voice Purification
Speaker Verification Attack
EnCodec Latent Space
Whisper-guided
🔎 Similar Papers
No similar papers found.
