Self-Guidance: Enhancing Neural Codecs via Decoder Manifold Alignment

2026-06-11

AI Summary

This work addresses the degradation in reconstruction fidelity of neural speech codecs caused by quantization error, a limitation often exacerbated by existing enhancement methods that increase the complexity of downstream language modeling. The authors propose a lightweight, self-guided feature manifold alignment mechanism that improves reconstruction quality without modifying the quantizer or expanding model capacity. By aligning the internal feature manifolds of quantized tokens and original continuous embeddings within the decoder, the method introduces a feature mapping loss to enhance fidelity. Implemented within a VQ-VAE framework, this approach enables efficient training and inference in codecs such as XCodec2, achieving state-of-the-art reconstruction performance at low bitrates, supporting high-fidelity synthesis under 4× codebook compression, and significantly improving LLM-driven text-to-speech generation.

Abstract

Neural speech codecs based on Vector-Quantized VAEs (VQ-VAEs) are core audio tokenizers for speech LLMs, yet their reconstruction fidelity is bottlenecked by quantization error. Modifying the quantizer or increasing model capacity are common fixes, but they complicate downstream language modeling. Our core idea is to align the decoder's internal feature manifolds when processing both the quantized tokens and their original continuous embeddings, using a lightweight feature-mapping loss. This requires minimal training overhead and no inference-time changes. Applied to XCodec2, self-guidance improves all reconstruction metrics, achieving state-of-the-art low-bitrate performance. Notably, it enables a 4x codebook reduction without fidelity loss, which downstream TTS experiments show significantly improves LLM-based synthesis by simplifying the token modeling space. Multiple statistical observations and visualizations corroborate the enhanced internal manifold alignment in the decoder. Extensive experiments confirm its generality across various inductive biases. Self-guidance thus establishes an efficient, broadly applicable method for high-fidelity neural audio coding.

Problem

• neural speech codecs
• quantization error
• reconstruction fidelity
• VQ-VAE
• audio tokenization

Innovation

• Self-Guidance
• Manifold Alignment
• Neural Speech Codec
• VQ-VAE
• Audio Tokenization