Experience-Calibrated Contrastive Decoding for Mitigating Hallucinations in LM-Based Text-to-Speech

2026-08-01

AI Summary

This work addresses the issue of speech hallucination in language model–driven text-to-speech synthesis, where generated audio often deviates from the input text due to overreliance on acoustic priors. Existing approaches typically require architectural modifications or additional training, lacking effective control during decoding. The paper proposes a training-free contrastive decoding method that, for the first time, introduces a conditional information perspective at inference time. By dynamically contrasting language model predictions with and without textual conditioning, the approach disentangles alignment-relevant signals from prior-driven artifacts while preserving the original acoustic distribution. Evaluated across four models, the method substantially reduces hallucination, achieving up to a 55.6% relative reduction in word/character error rates and a +0.644 CMOS gain in subjective listening tests, all while maintaining high naturalness and speaker similarity.

Abstract

Language model-based text-to-speech (LM-based TTS) remains vulnerable to speech hallucinations that deviate from the target text. Existing mitigation mainly relies on architectural changes or additional training, while decoding-time control remains underexplored. We present a conditional information view that distinguishes text-derived alignment information from experience information supplied by acoustic context and learned speech regularities. We hypothesize that an important class of hallucinations begins when alignment support is insufficiently reflected in the selected token at a vulnerable transition. Using predictions from the same speech LM with and without text conditions, we propose Experience-Calibrated Contrastive Decoding (ECCD), a training-free method that strengthens alignment support while preserving useful experience information. ECCD preserves the original expert distribution, applies only positive alignment enhancement, and calibrates its strength using set-level experience compatibility. Across four models, ECCD reduces WER/CER by up to 55.6% in all SeedTTS-Eval settings and 24 of 25 multilingual CV3-Eval settings. A listening test yields a CMOS gain of $+0.644$ while retaining strong speaker similarity. Further analysis shows that alignment influence and decision-level gain vary within linguistic units and are lower at first-error boundaries than at matched correct boundaries. Overall, these extensive experiments and analyses identify conditional information control as a promising decoding-time direction for mitigating speech hallucination.

Problem

• speech hallucination
• LM-based TTS
• decoding-time control
• text-to-speech
• alignment

Innovation

• contrastive decoding
• speech hallucination
• alignment calibration
• training-free mitigation
• conditional information control