LLM-Codec: Neural Audio Codec Meets Language Model Objectives

2026-04-20

AI Summary

This work addresses the mismatch between the optimization objectives of existing neural audio codecs and the autoregressive prediction requirements of language models, which introduces acoustic uncertainty in discrete tokens and increases language model perplexity. Without altering the codec or large language model architectures, the authors propose a language-model-oriented joint optimization framework that enhances token predictability through a Medusa-style multi-step future token prediction head, aligns audio-text semantics via a memory-bank-based contrastive loss, and enables end-to-end training using a differentiable Gumbel bridge. Experiments demonstrate that the proposed method achieves a 61.6% accuracy (+12.1%) on the SALMon speech coherence task with a 35-point reduction in perplexity, and improves Mel-distance by 5.0% on Codec-SUPERB-tiny while preserving both reconstruction fidelity and token learnability.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Neural audio codecs are widely used as tokenizers for spoken language models, but they are optimized for waveform reconstruction rather than autoregressive prediction. This mismatch injects acoustically driven uncertainty into the discrete token space and increases language-model perplexity. We propose \ours, which augments codec training with language-model-facing objectives while keeping both codec and LLM architectures unchanged. \ours introduces (i) future token prediction with Medusa-style multi-step heads to encourage multi-step predictability, and (ii) semantic alignment that matches audio and text representations via a memory-bank contrastive loss. A differentiable Gumbel bridge enables end-to-end gradients from these objectives to the codec encoder. On SALMon speech coherence, token LMs trained on \ours reach 61.6% accuracy (+12.1 points over AUV) while reducing perplexity 35. On Codec-SUPERB-tiny, \ours improves speech Mel distance by 5.0% over AUV while simultaneously achieving the learnability gains, demonstrating that reconstruction fidelity and token predictability can be improved together.

Problem

• neural audio codec
• language model
• tokenization
• autoregressive prediction
• perplexity

Innovation

• neural audio codec
• language model objectives
• multi-step prediction
• semantic alignment
• Gumbel bridge