Listen, Critique, and Refine: RL-Based Self-Refinement for Instruction-Following Speech Synthesis

2026-09-21

AI Summary

为解决复杂语音合成指令的实现问题，通过强化学习训练大型音频语言模型进行自我反思和优化，提升合成质量。

Abstract

Large Audio Language Models (LALMs) can follow diverse instructions to synthesize speech in specified styles. However, complex instructions that require simultaneous control over pitch dynamics, speaking rate, and emotional tone often exceed what a single-pass generation can faithfully realize. While recent reasoning models have shown that intermediate "thinking" tokens improve output quality, this paradigm has been confined to the text modality. In this work, we extend reasoning to the audio token space by training a LALM with reinforcement learning to reason over its own speech output. The model first generates a draft speech as a form of audio-token reasoning, critiques its own generation by reflecting on the acoustic realization in text, and then produces a refined version conditioned on both the first-pass speech and the critique, all within a single model. After RL training, the refined two-hop outputs achieve a relative improvement of 7.15\% on the InstructTTSEval benchmark, demonstrating the model's reflective ability.

Problem

• Large Audio Language Models
• complex instructions
• speech synthesis

Innovation

• Reinforcement Learning
• Self-Refinement
• Audio Token Reasoning
• Instruction-Following Speech Synthesis