Hierarchical Control of Emotion Rendering in Speech Synthesis

2024-12-17

AI Summary

Fine-grained, quantifiable control of multi-level emotional intensity remains challenging in expressive speech synthesis. Method: This paper proposes a hierarchical emotion rendering framework based on diffusion models, enabling controllable synthesis across phoneme-, word-, and sentence-level emotion intensities. We introduce the first Hierarchical Emotion Distribution (ED) extractor, which explicitly models quantifiable emotion embeddings across speech segments and captures their correlations with linguistic and speaker characteristics. The method integrates reference-audio-driven emotion embedding learning, hierarchical feature encoding, and multi-scale acoustic analysis—including F0, energy, duration, and Mel-spectral dynamics. Results: Experiments demonstrate significant improvements over baselines in both objective metrics (e.g., MCD, F0 RMSE) and subjective evaluations (e.g., MOS, Emo-MOS, emotion control accuracy), achieving fine-grained, adjustable, and high-fidelity emotional speech generation.

Technology Category

• Natural Language Processing: Speech
• Cognitive Modeling & Cognitive Systems: Affective Computing
• Humans and AI: Emotional Intelligence

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Emotional text-to-speech synthesis (TTS) aims to generate realistic emotional speech from input text. However, quantitatively controlling multi-level emotion rendering remains challenging. In this paper, we propose a diffusion-based emotional TTS framework with a novel approach for emotion intensity modeling to facilitate fine-grained control over emotion rendering at the phoneme, word, and utterance levels. We introduce a hierarchical emotion distribution (ED) extractor that captures a quantifiable ED embedding across different speech segment levels. Additionally, we explore various acoustic features and assess their impact on emotion intensity modeling. During TTS training, the hierarchical ED embedding effectively captures the variance in emotion intensity from the reference audio and correlates it with linguistic and speaker information. The TTS model not only generates emotional speech during inference, but also quantitatively controls the emotion rendering over the speech constituents. Both objective and subjective evaluations demonstrate the effectiveness of our framework in terms of speech quality, emotional expressiveness, and hierarchical emotion control.

Problem

• Emotional Expression
• Text-to-Speech
• Computer-Generated Voice

Innovation

• Emotion Intensity Control
• Multilevel Emotional Expression
• Voice Quality and Emotional Delivery