VibeVoice-ASR-BitNet Technical Report

2026-07-23

AI Summary

This work addresses real-time speech recognition on edge-device CPUs under stringent resource constraints, balancing model compression and accuracy. It proposes a heterogeneous quantization strategy: the acoustic encoder employs full INT8 pipeline quantization with SIMD optimization, while the language model adopts BitNet-inspired ternary weights (I2_S), complemented by progressive quantization-aware training to mitigate accuracy degradation. Built upon the ggml framework, the system features custom high-efficiency kernels for ARM and x86 platforms, achieving real-time inference with RTF < 1 using only three CPU threads—yielding a 1.6–2.3× speedup over Whisper.cpp. The resulting model occupies approximately 1.6 GB of storage with well-controlled accuracy loss.

Abstract

We present VibeVoice-ASR-BitNet, a compressed variant of VibeVoice-ASR optimized for real-time inference on edge CPUs. We apply heterogeneous quantization tailored to the computational characteristics of each stage: the VAE acoustic tokenizer uses full-pipeline INT8 quantization (I8_S) with kernel fusion and SIMD optimization, while the autoregressive language model adopts BitNet-style ternary weights (I2_S). To preserve accuracy under aggressive compression, we employ a progressive quantization-aware training strategy. For inference, we implement custom SIMD kernels and fused operators within the ggml framework targeting both ARM and x86 platforms, achieving real-time recognition with RTF < 1 using as few as 3 CPU threads. VibeVoice-ASR-BitNet is 1.6-2.3x faster than Whisper.cpp at comparable model sizes (~1.6 GB), with only modest accuracy degradation compared to the FP16 baseline.

Problem

• automatic speech recognition
• edge computing
• model compression
• real-time inference
• quantization

Innovation

• heterogeneous quantization
• BitNet-style ternary weights
• quantization-aware training
• SIMD optimization
• edge ASR