🤖 AI Summary
To address GPU memory constraints and inefficient batching caused by variable-length inputs in industrial-scale long-audio automatic speech recognition (ASR), this paper proposes a mask-based chunked batching mechanism and a relative right-context modeling approach tailored for the Conformer architecture. It enables, for the first time, padding-free dynamic batching and end-to-end modeling of hour-long speech. Our method supports transcription of audio up to 16 hours on an 80 GB GPU—1.5× longer than FastConformer’s maximum. It reduces word error rate (WER) by 7.7 percentage points absolutely, cuts batch memory consumption and latency by over 3×, and matches the original Conformer’s accuracy on short utterances. The core innovations lie in the synergistic optimization of chunked self-attention, relative positional encoding, and masked chunking—effectively mitigating padding redundancy and extending Conformer’s capability to ultra-long speech with minimal memory overhead.
📝 Abstract
Deploying ASR models at an industrial scale poses significant challenges in hardware resource management, especially for long-form transcription tasks where audio may last for hours. Large Conformer models, despite their capabilities, are limited to processing only 15 minutes of audio on an 80GB GPU. Furthermore, variable input lengths worsen inefficiencies, as standard batching leads to excessive padding, increasing resource consumption and execution time. To address this, we introduce ChunkFormer, an efficient ASR model that uses chunk-wise processing with relative right context, enabling long audio transcriptions on low-memory GPUs. ChunkFormer handles up to 16 hours of audio on an 80GB GPU, 1.5x longer than the current state-of-the-art FastConformer, while also boosting long-form transcription performance with up to 7.7% absolute reduction on word error rate and maintaining accuracy on shorter tasks compared to Conformer. By eliminating the need for padding in standard batching, ChunkFormer's masked batching technique reduces execution time and memory usage by more than 3x in batch processing, substantially reducing costs for a wide range of ASR systems, particularly regarding GPU resources for models serving in real-world applications.