GROW: Group-Relative Advantage-Weighted On-Policy Reinforcement Learning of Autoregressive-Diffusion Text-to-Speech model

2026-08-04

AI Summary

This work addresses the challenges in reinforcement learning for flow-matching text-to-speech synthesis, where deterministic ODE sampling complicates policy gradient estimation, necessitates stochastic perturbations, and incurs high computational costs. To overcome these issues, the authors propose GROW, a method that performs direct policy optimization on the standard flow-matching objective without converting the ODE to an SDE. GROW introduces a novel group-relative advantage weighting mechanism and zero-mean signed advantages to enable effective intra-group credit assignment and mitigate self-imitation dominance. It further integrates intra-group reward normalization, advantage-weighted regression, and Wasserstein-2 velocity regularization, while freezing the pretrained model as a fixed reference anchor. Experiments demonstrate that GROW reduces WER from 2.016 to 1.558 and improves speaker similarity from 0.676 to 0.715 on LibriSpeech and Seed-TTS, achieving a 2.9× faster training speed than DiTAR-GRPO while maintaining UTMOS scores.

Abstract

Reinforcement learning for flow-matching text-to-speech is complicated by deterministic ODE sampling: trajectory-level policy-gradient methods typically convert the ODE into an SDE and track per-step likelihood ratios, introducing stochastic perturbations and substantial overhead. We propose GROW, a group-relative advantage-weighted on-policy RL method that acts directly on the standard flow-matching objective. For each prompt, GROW samples a group of on-policy utterances, separately standardizes intelligibility and speaker-similarity rewards within the group, and combines them to reweight flow-matching regression. A Wasserstein-2 velocity penalty anchors the updated model to a frozen pretrained reference. A group-mean reward baseline is introduced to convert reward weighting into advantage weighting. For strong pretrained TTS models with concentrated rewards, positive exponential weighting is dominated by reward-agnostic self-imitation, whereas a zero-mean signed advantage preserves effective within-group credit assignment. Instantiated on DiTAR and evaluated on LibriSpeech and Seed-TTS EN/ZH, GROW reduces average WER from 2.016 to 1.558 and raises speaker similarity from 0.676 to 0.715 while keeping UTMOS. With 10-NFE training rollouts and 32-NFE evaluation, GROW retains comparable performance while training 2.9x faster than 32-NFE DiTAR-GRPO. We will open-source complete GROW codes, faithful DiTAR reproduction, and all model checkpoints.

Problem

• reinforcement learning
• text-to-speech
• flow-matching
• ODE sampling
• policy gradient

Innovation

• group-relative advantage weighting
• on-policy reinforcement learning
• flow-matching TTS
• Wasserstein-2 velocity penalty
• reward standardization