Maestro-EVC: Controllable Emotional Voice Conversion Guided by References and Explicit Prosody

2025-08-09

AI Summary

Existing emotional voice conversion (EVC) methods struggle to disentangle speaker identity from fine-grained emotional dynamics—such as temporal prosodic variations—resulting in poor controllability and limited expressiveness. To address this, we propose an end-to-end disentangled framework: (1) independent reference signals are introduced to separately control linguistic content, speaker identity, and emotional style; (2) a temporal emotion encoder is designed to explicitly model the evolution of emotional states over time; and (3) explicit prosody modeling is integrated with prosody enhancement strategies to improve robustness across diverse prosodic scenarios. Experiments demonstrate that our method achieves significantly improved independent control over the three attributes—content, speaker, and emotion—while preserving high naturalness. It also enhances emotional expressiveness substantially, outperforming state-of-the-art approaches in both objective metrics and subjective listening tests.

Technology Category

• Natural Language Processing: Speech
• Humans and AI: Emotional Intelligence
• Cognitive Modeling & Cognitive Systems: Affective Computing

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search

Abstract

Emotional voice conversion (EVC) aims to modify the emotional style of speech while preserving its linguistic content. In practical EVC, controllability, the ability to independently control speaker identity and emotional style using distinct references, is crucial. However, existing methods often struggle to fully disentangle these attributes and lack the ability to model fine-grained emotional expressions such as temporal dynamics. We propose Maestro-EVC, a controllable EVC framework that enables independent control of content, speaker identity, and emotion by effectively disentangling each attribute from separate references. We further introduce a temporal emotion representation and an explicit prosody modeling with prosody augmentation to robustly capture and transfer the temporal dynamics of the target emotion, even under prosody-mismatched conditions. Experimental results confirm that Maestro-EVC achieves high-quality, controllable, and emotionally expressive speech synthesis.

Problem

• Disentangle speaker identity and emotional style in voice conversion
• Model fine-grained emotional expressions like temporal dynamics
• Achieve controllable emotional voice conversion under prosody-mismatched conditions

Innovation

• Disentangles content, speaker, and emotion attributes
• Uses temporal emotion representation for dynamics
• Explicit prosody modeling with augmentation