Psychoacoustically Aligned Latent Smoothing for Adversarial Robustness of Full-Duplex Speech-to-Speech Dialogue Models

2026-09-23

AI Summary

研究针对全双工语音对话模型的对抗攻击问题，提出基于心理声学对齐的潜在平滑方法(PALS)以增强模型鲁棒性。

Abstract

End-to-end speech-to-speech dialogue models listen and speak simultaneously, so a continuously open acoustic channel is exposed to adversarial manipulation. We formalize imperceptible attacks on full-duplex agents as optimization over additive perturbations confined beneath the psychoacoustic masking threshold of the carrier speech, under three goals: targeted semantic hijacking, response suppression, and policy jailbreaking. Against an undefended Moshi-style agent, white-box attacks succeed in up to 91.7% of trials. We then introduce psychoacoustically aligned latent smoothing (PALS), which injects anisotropic Gaussian noise shaped by local codebook covariance at the residual-vector-quantized latent interface, with input noise shaped by the masking threshold constraining the attacker and trained by a Kullback--Leibler consistency objective. Deployed with no inference-time cost, PALS reduces hijack to 8.3%, mute to 11.2%, and jailbreak to 9.1% at clean quality within 2.3%. A Monte Carlo-smoothed variant certifies an ellipsoidal latent radius up to 0.616, a guaranteed floor that the empirical robustness far exceeds.

Problem

• adversarial manipulation
• psychoacoustic masking threshold
• speech-to-speech dialogue models

Innovation

• Psychoacoustically Aligned Latent Smoothing (PALS)
• Adversarial Robustness
• Full-Duplex Speech-to-Speech Dialogue Models
• Anisotropic Gaussian Noise