From Script to Drama: An Agentic Framework for Controllable Multi-Speaker Dialogue TTS

2026-09-27

AI Summary

This study addresses the challenge of balancing naturalness, speaker identity consistency, and fine-grained emotional control in multi-speaker text-to-speech (TTS) for long conversations. We propose a critique-driven iterative optimization framework that employs ControlEdit-TTS as the backbone network and integrates an agent-based workflow to unify speech synthesis into instruction-following and attribute-editing tasks. A key innovation is the introduction of hierarchical utterance- and scene-level critique mechanisms, enabling precise error correction and temporal adjustment without requiring full regeneration. Experimental results on Chinese-English bilingual benchmarks demonstrate that our approach significantly improves instruction adherence and conversational preference, outperforming both direct generation and regeneration-only baselines.

Abstract

Multi-speaker dialogue TTS requires natural speech generation, consistent speaker identity, coherent cross-turn transitions, and fine-grained control of expressive attributes such as emotion, speaking rate, and loudness. These requirements are difficult to satisfy reliably with one-shot generation, especially in long-form dialogue. We propose a controllable multi-speaker dialogue TTS framework that formulates synthesis as critique-driven iterative refinement. Its speech backbone, ControlEdit-TTS, unifies instruction-following synthesis and natural-language-guided attribute editing, enabling correction of expressive errors without full regeneration. The framework further performs hierarchical utterance-level and scene-level critique, routing detected issues to editing, resynthesis, or timing adjustment. Experiments on a bilingual Chinese--English dialogue benchmark show improved utterance-level instruction following, better dialogue-level preference than direct dialogue models and agentic baselines, and more effective refinement than regeneration-only alternatives while preserving speaker identity. Ablations further confirm the benefits of scene-level critique and edit-based correction.

Problem

• Multi-speaker dialogue TTS
• Controllable speech synthesis
• Expressive attribute control
• Speaker identity consistency

Innovation

• Multi-speaker dialogue TTS
• Iterative refinement
• Controllable speech synthesis
• Attribute editing
• Agentic framework