Problem
Research questions and friction points this paper is trying to address.
Automatic Speech Recognition
Conformer
Integer-only quantization
Mobile NPU
Edge deployment
I-Parakeet: Integer-Only Conformer ASR on Mobile NPU
🤖 AI Summary
This study addresses the acceleration bottleneck caused by floating-point fallback when deploying large Conformer models on mobile NPUs, proposing a full integer quantization scheme tailored for Parakeet CTC models. Core innovations include the first derivation of fully integer arithmetic formulations for relative positional self-attention and the design of a minimax Swish approximation strategy. Combined with operator fusion, INT16 grid optimization, and percentile-based activation calibration, the method achieves deep adaptation to Qualcomm NPUs. This approach completely eliminates CPU fallback, fully unleashing NPU computational capacity. Experiments demonstrate that the quantized model attains a word error rate of 4.97% on the LibriSpeech test set with a real-time factor of 0.048, yielding a 7.5× speedup over the CPU baseline and achieving an excellent balance between accuracy and on-device inference efficiency.
📝 Abstract
In this paper, we propose I-Parakeet, an integer-only implementation of NVIDIA's Parakeet-CTC (0.6B parameters) that runs on a smartphone NPU without any floating-point operator or CPU fallback. Modern Conformer ASR models are hard to deploy on edge devices because of their size, and quantized models still fall back to floating point for numerically sensitive operations. This prevents them from fully exploiting integer accelerators such as mobile NPUs. To achieve this, our contributions are threefold. First, we derive an integer formulation of the relative-positional self-attention at the core of the Conformer. We fuse its two score branches with different quantization scales and the relative shift into integer-only operations. Second, we introduce a minimax-optimized Swish approximation that minimizes the maximum error of the Swish output. Third, a layer-wise range analysis of activations yields two targeted remedies: an INT16 grid for the BatchNorm output and percentile calibration for the heavy-tailed pre-encoder activations. I-Parakeet achieves 4.97% WER on LibriSpeech test-other, running on a Qualcomm NPU at a real-time factor of 0.048, 7.5x faster than a CPU baseline.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Integer-Only Quantization
Conformer ASR
Mobile NPU
Relative-Positional Self-Attention
Minimax Swish Approximation
🔎 Similar Papers
No similar papers found.
