Stable Autoregressive Speech Generation with Low-Frame-Rate High-Dimensional Continuous Tokens

2026-07-31

AI Summary

This work addresses the challenges in autoregressive speech generation, where high frame rates or high-dimensional representations often lead to distributional drift and error accumulation, while low-dimensional representations compromise reconstruction fidelity. To overcome these limitations, the authors propose a jointly optimized framework featuring a low frame rate (8 Hz) yet high-dimensional (768-dim) continuous speech representation and a streaming generation architecture. Central to this approach is Locodec, a locally conditioned codec that enhances representational interpolability and coordinate identifiability, coupled with MP-ELD—a single-token autoregressive flow-matching mechanism incorporating multi-path routing and residual classifier-free guidance to effectively mitigate error propagation. Notably, the method achieves competitive word error rates (WER) and long-term stable, high-fidelity synthesis without relying on external SSL/ASR models, pretrained language models, or post-training, while maintaining high reconstruction quality and strong single-token predictability.

Abstract

Balancing sequence length, representational capacity, and long-horizon stability is a central problem in autoregressive (AR) speech and audio generation. Representations with higher frame rates or greater capacity can preserve more signal detail, but they also make streaming generation more vulnerable to distribution drift and AR error accumulation. Conversely, shorter and more compressed representations simplify AR modeling, but their limited bandwidth may discard important components and constrain the upper bound of reconstruction fidelity and generation quality. We ask whether a low-frame-rate, high-dimensional, high-bandwidth continuous representation can be co-designed with a streaming generation framework to support robust high-fidelity reconstruction, strong single-token predictability, and superior long-horizon stability. We decompose this goal into two coupled problems: what geometric and statistical properties a high-dimensional representation space should have, and how an AR continuous-token generator should be structured to resist error accumulation. Accordingly, we propose Locodec, a locally encoded codec that shapes its representation space to improve the interpolatability of a lower-dimensional core manifold and the identifiability of the native high-dimensional coordinates, thereby improving the predictability of high-dimensional high-bandwidth tokens. We also propose MP-ELD, a single-token AR flow-matching framework that uses multi-path information routing and residual classifier-free guidance to mitigate error accumulation. Experiments with 8-Hz, 768-dimensional tokens show that our design preserves reconstruction quality, improves single-token predictability, achieves competitive WER, and maintains stable long-form synthesis, without using external SSL/ASR models, pretrained text language models, or post-training stages.

Problem

• autoregressive speech generation
• representation stability
• error accumulation
• high-dimensional tokens
• long-horizon synthesis

Innovation

• continuous tokens
• autoregressive speech generation
• low-frame-rate representation
• error accumulation mitigation
• high-dimensional manifold