Multi-Step Prediction and Control of Hierarchical Emotion Distribution in Text-to-Speech Synthesis

2025-07-06

AI Summary

This work addresses the challenge of quantitatively modeling and disentangling multi-level emotions (utterance-, word-, and phoneme-level) in text-to-speech (TTS). We propose a multi-step hierarchical emotion distribution (ED) prediction framework. Methodologically, we design a hierarchical variance prediction module and a multi-step inference strategy, incorporating global emotional context to constrain local emotional variations, and introduce a variance adapter for plug-and-play integration with mainstream TTS architectures. To our knowledge, this is the first approach enabling fine-grained, cross-granularity emotional rendering with independent control over emotion intensity at each level. Objective evaluations and subjective MOS tests demonstrate significant improvements in emotional expressiveness, outperforming existing baselines in both semantic consistency and emotional naturalness. The framework supports interpretable, fine-grained intervention on emotion intensity.

Technology Category

• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Humans and AI: Emotional Intelligence
• Machine Learning: Multimodal Learning

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

We investigate hierarchical emotion distribution (ED) for achieving multi-level quantitative control of emotion rendering in text-to-speech synthesis (TTS). We introduce a novel multi-step hierarchical ED prediction module that quantifies emotion variance at the utterance, word, and phoneme levels. By predicting emotion variance in a multi-step manner, we leverage global emotional context to refine local emotional variations, thereby capturing the intrinsic hierarchical structure of speech emotion. Our approach is validated through its integration into a variance adaptor and an external module design compatible with various TTS systems. Both objective and subjective evaluations demonstrate that the proposed framework significantly enhances emotional expressiveness and enables precise control of emotion rendering across multiple speech granularities.

Problem

• Achieve multi-level emotional control in TTS synthesis
• Predict hierarchical emotion variance at utterance, word, phoneme levels
• Enhance emotional expressiveness and precise rendering in speech

Innovation

• Multi-step hierarchical ED prediction module
• Global context refines local emotional variations
• Compatible with various TTS systems