ParaASR: Multi-Token Prediction for Fast and Long-Context LLM-Based Speech Recognition

2026-07-31

AI Summary

This work addresses the longstanding challenge in automatic speech recognition (ASR) of simultaneously achieving high accuracy, low latency, and effective long-context modeling within large language model–based systems. The authors propose ParaASR, the first framework to effectively integrate multi-token prediction into ASR by leveraging the strong constraints inherent in speech-to-text mapping. Their 4B-parameter decoder generates up to six tokens in a single forward pass through a non-autoregressive main branch and five future-token branches, followed by an autoregressive verification step to ensure transcription fidelity. Built upon an audio–language foundation model and trained via a staged strategy, ParaASR achieves word error rates of 2.97%, 3.68%, and 3.70% on English, Chinese, and long-form audio benchmarks, respectively, with a remarkably low real-time factor of 0.0053, while supporting 32K-token context and transcribing up to 30-minute audio segments in one pass.

Abstract

Audio-encoder-LLM-decoder architectures have become the dominant paradigm for modern automatic speech recognition (ASR), improving transcription quality through large-scale language modeling. However, the cost of autoregressive decoding scales with decoder size, creating a fundamental trade-off between recognition quality and serving latency. We argue this trade-off is not inherent: unlike open-ended text generation, ASR outputs are strongly anchored to the input speech signal, providing a natural inductive bias toward high-parallelism decoding. Building on this, we introduce ParaASR, an ASR system that leverages Multi-Token Prediction (MTP) to let a 4B LLM decoder emit multiple tokens per forward step. Starting from a publicly available audio-language foundation, the model first establishes a robust autoregressive recognizer and then aligns five future-token branches through a staged optimization recipe. At inference, it proposes a six-token continuation per step and admits only the verified prefix into the transcript, preserving the safety of standard autoregressive decoding. The average accepted length reaches 5.0 out of 6 proposed tokens, confirming that the deterministic structure of speech makes ASR an especially natural setting for multi-token decoding. ParaASR further retains a native 32K-context window and transcribes up to 30 minutes of audio in a single pass. Across diverse benchmarks, it attains average error rates of 2.97%, 3.68%, and 3.70% on Chinese, English, and long-form evaluations, respectively, while reaching a real-time factor (RTF) as low as 0.0053. These results show that decoder scaling, low-latency inference, and long-context transcription need not be competing goals when future-token proposals are anchored by the acoustic signal and guarded by autoregressive verification.

Problem

• automatic speech recognition
• autoregressive decoding
• latency-quality trade-off
• long-context transcription
• multi-token prediction

Innovation

• Multi-Token Prediction
• Parallel Decoding
• LLM-based ASR
• Long-Context Speech Recognition
• Autoregressive Verification