Stabilizing Instruction Supervision for Instruct-TTS via Controllable Diversification and Drift Filtering

2026-09-08

AI Summary

该研究针对Instruct-TTS系统中由语义漂移引起的问题，提出通过可控多样化、漂移过滤及属性对齐监督的方法提高指令的稳定性和质量。

Abstract

Instruct-TTS systems expand structured style labels into natural-language training instructions through LLM rewriting, yet we find that over 40% of unconstrained rewrites contain semantic drift that corrupts supervision and weakens generalization. We formalize this problem as instruction supervision instability and propose a data-centric stabilization recipe that jointly improves coverage and fidelity through three mechanisms: controllable instruction diversification for systematic expansion, LLM-based drift filtering for quality control, and attribute-aligned supervision that grounds prosody control in acoustic perturbations. On the Chinese split of InstructTTSEval, our recipe raises instruction-following from 34.5% without fine-tuning and 51.0% with naive fine-tuning to 56.4%, while constrained rewriting reduces drift from 40.4% to 15.4%. Ablations confirm the three mechanisms are complementary, and the drift taxonomy may generalize to instruction-driven generation beyond TTS.

Problem

• Instruction Supervision
• Semantic Drift
• TTS
• LLM Rewriting
• Generalization

Innovation

• controllable diversification
• drift filtering
• attribute-aligned supervision