End-to-End Training for Discrete Token LLM based TTS System

2026-06-08

AI Summary

This work addresses the inference mismatch between modules in conventional discrete speech token-based text-to-speech (TTS) systems, which arises from cascaded training. The authors propose the first end-to-end joint optimization framework that unifies the training of a speech tokenizer, an autoregressive large language model, a flow-matching acoustic model, and a reward model. By integrating acoustic reconstruction and semantic recognition objectives through multi-task learning, the approach substantially simplifies the training pipeline. Evaluated on the Seed-TTS-Eval benchmark, the method achieves new state-of-the-art results with word error rates of 0.78% and 1.56%, demonstrating the effectiveness and superiority of end-to-end optimization in discrete-token TTS systems.

Abstract

Recent state-of-the-art (SOTA) text-to-speech (TTS) systems typically adopt a cascaded pipeline consisting of a speech tokenizer, an autoregressive large language model (LLM), and a diffusion based flow-matching (FM) model, with these components trained independently. In this paper, we propose a fully end-to-end (E2E) optimization framework that unifies the training of the speech tokenizer, LLM, FM model, and an additional reward model (RM). Specifically, we first jointly optimize the tokenizer using multi-task objectives derived from reconstruction for FM, next-token prediction for LLM, and multi recognition task for RM. This joint training encourages the discrete speech token space to capture acoustically and semantically salient information that is better tailored to TTS. We then further optimize the LLM using downstream reconstruction and recognition by FM and RM, which reduces inference-time mismatch and steers the LLM toward more preferred generations. Experimental results show that our E2E framework consistently outperforms cascaded baselines. On the Seed-TTS-Eval benchmark, our system achieves a word error rate (WER) of 0.78% and 1.56%, a new SOTA result with a 0.6B-parameter LLM and 0.5B-parameter FM model. These results validate that holistic E2E optimization is critical for improving discrete-token-based TTS systems with a much simpler training pipeline.

Problem

• text-to-speech
• discrete token
• end-to-end training
• large language model
• speech tokenizer

Innovation

• end-to-end training
• discrete token TTS
• speech tokenizer
• large language model
• flow-matching