Optimality of FSQ Tokens for Continuous Diffusion for Categorical Data with Application to Text-to-Speech

2026-06-08

AI Summary

This work investigates the optimal discrete tokenization strategy for continuous diffusion models to efficiently generate categorical data such as speech. Through theoretical analysis and empirical validation, the study establishes—via KL divergence and optimal prediction criteria—that Finite Scalar Quantization (FSQ) induces an optimal latent space structure within the continuous diffusion framework. Building upon this insight, the authors develop an FSQ-driven, non-autoregressive diffusion-based text-to-speech (TTS) model. This model significantly outperforms alternative tokenization schemes in speech token modeling, surpasses larger language model baselines despite its smaller parameter count, and achieves faster inference speeds.

Abstract

Continuous diffusion for categorical data is a framework belonging to the diffusion family and aiming at generating discrete data. The scientific interest to such models has been constantly increasing these days because researchers try to achieve a challenging goal of finding reasonable alternatives to autoregressive large language models. In this paper, we study the properties of the structure of the latent space corresponding to discrete tokens expressed in terms of Kullback-Leibler divergence on diffusion path measures and accuracy of the correct token prediction by the optimally trained diffusion model. We find that FSQ tokenization scheme has the latent space structure with the properties that make it best suited for continuous diffusion for categorical data as verified through rigorous theoretical analysis and numerical experiments. To validate our findings in real-life scenario, we train several text-to-speech diffusion models having speech tokens as intermediate acoustic features, and show that the one based on FSQ tokens indeed performs the best, and, moreover, it outperforms its strong LLM-based counterpart, at the same time being significantly smaller and faster.

Problem

• continuous diffusion
• categorical data
• tokenization
• latent space
• text-to-speech

Innovation

• FSQ tokenization
• continuous diffusion
• categorical data generation
• latent space structure
• text-to-speech