Improving Robustness of Diffusion-Based Zero-Shot Speech Synthesis via Stable Formant Generation

2024-09-14

arXiv.org

AI Summary

Diffusion-based zero-shot text-to-speech (TTS) suffers from unstable formants and phoneme mispronunciations due to the absence of articulatory physical constraints. To address this, we propose StableForm-TTS—the first diffusion TTS framework incorporating source-filter theory to enforce biophysically grounded articulatory constraints for stable formant modeling. Methodologically, we design an acoustically decoupled conditional diffusion model, introduce a formant-aware loss, and employ hierarchical time-frequency feature guidance; further, we adopt a speaker-agnostic zero-shot adaptation strategy. Experiments demonstrate that StableForm-TTS improves phoneme accuracy by 12.6% on unseen speakers, increases naturalness (MOS) by 0.42, and maintains high speaker similarity. Moreover, performance consistently scales with both data volume and model size, confirming its robustness and extensibility.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Speech
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Diffusion models have achieved remarkable success in text-to-speech (TTS), even in zero-shot scenarios. Recent efforts aim to address the trade-off between inference speed and sound quality, often considered the primary drawback of diffusion models. However, we find a critical mispronunciation issue is being overlooked. Our preliminary study reveals the unstable pronunciation resulting from the diffusion process. Based on this observation, we introduce StableForm-TTS, a novel zero-shot speech synthesis framework designed to produce robust pronunciation while maintaining the advantages of diffusion modeling. By pioneering the adoption of source-filter theory in diffusion TTS, we propose an elaborate architecture for stable formant generation. Experimental results on unseen speakers show that our model outperforms the state-of-the-art method in terms of pronunciation accuracy and naturalness, with comparable speaker similarity. Moreover, our model demonstrates effective scalability as both data and model sizes increase. Audio samples are available online: https://deepbrainai-research.github.io/stableformtts/.

Problem

• Text-to-Speech
• Stability and Accuracy
• Diffusion Models

Innovation

• StableForm-TTS
• Enhanced Acoustic Feature Stability
• Diffusion Model Optimization