Audio-Native Speech Recognition with a Frozen Discrete-Diffusion Language Model

2026-07-14

AI Summary

This work addresses the inherent inefficiency of autoregressive speech recognition in parallelization by proposing a parallel speech transcription method based on the frozen discrete diffusion language model DiffusionGemma. The approach integrates a frozen Whisper encoder, a lightweight cross-lingual projector, and low-rank adapters, training only 0.16% of the total parameters. Full transcriptions are generated in approximately eight parallel denoising steps. The study introduces, for the first time in speech recognition, a uniform random token-based discrete diffusion mechanism and incorporates a CTC loss to mitigate gradient vanishing issues arising from model freezing. Evaluated on LibriSpeech test-clean, the method achieves a word error rate of 6.6%. Furthermore, a single multilingual adapter demonstrates effective performance across English, Hindi, and Chinese.

Abstract

Automatic speech recognition is dominated by autoregressive decoders that emit one token at a time. We ask whether a discrete diffusion language model can transcribe speech instead, refining a whole transcript in parallel over a small number of denoising steps. We train an audio-native interface for DiffusionGemma, a 26B mixture-of-experts model that generates text by uniform, random-token discrete diffusion rather than the absorbing-mask scheme common to recent diffusion language models. A frozen Whisper encoder supplies acoustic features, a lightweight projector maps them into the model embedding space, and low-rank adapters let the frozen backbone attend to the new modality. About 42M parameters are trained, which is 0.16 percent of the backbone. We find that the natural training objectives fail to ground the audio because their gradient reaches the projector only through attention that has already dismissed it. A connectionist temporal classification loss applied through the frozen output head breaks this deadlock. The resulting model reaches 6.6 percent word error rate on LibriSpeech test-clean, transcribes in roughly eight parallel steps regardless of utterance length, and uses a single adapter trained on six languages, which we evaluate here on English, Hindi, and Mandarin.

Problem

• speech recognition
• discrete diffusion
• non-autoregressive
• language model
• audio transcription

Innovation

• discrete diffusion language model
• frozen backbone
• parallel transcription
• low-rank adapters
• connectionist temporal classification