ViP-VL: Vietnamese Self-supervised Speech Pretraining Model with Vector-Quantization Learning

2026-06-08

AI Summary

This work addresses the limitations of existing self-supervised speech pretraining models for Vietnamese, which struggle to efficiently process high-resolution audio and exhibit insufficient generalization under limited computational resources. The authors propose ViP-VL, the first approach integrating vector-quantized learning with the ChunkFormer architecture within the BEST-RQ framework. By incorporating acoustic stacking, receptive field alignment, and a tailored masking strategy, ViP-VL achieves 8× synchronous downsampling while enhancing representation robustness. Pretrained on 17,000 hours of unlabeled Vietnamese speech, the model establishes state-of-the-art performance across four diverse downstream tasks—automatic speech recognition, speech emotion recognition, dialect classification, and speaker verification—demonstrating significantly improved multitask generalization capabilities.

Abstract

We present ViP-VL, an efficient Vietnamese Self-supervised speech Pretraining model leveraging Vector-quantization Learning. To bridge the gap between high-resolution audio and efficient processing, ViP-VL incorporates Acoustic Stacking and Receptive Field Alignment to enable a synchronized 8x subsampling rate within the ChunkFormer architecture, while further enhancing representation robustness through a specialized Mask Selection Strategy during pretraining on the BEST-RQ framework. Pretrained on 17,000 hours of unlabeled Vietnamese speech, our model establishes new state-of-the-art results across four major downstream tasks: Automatic Speech Recognition, Speech Emotion Recognition, Dialect Classification, and Speaker Verification. To facilitate future research and the development of high-performance Vietnamese speech technologies, we publicly release our pretrained weights and implementation at github.com/khanld/chunkformer.

Problem

• Vietnamese speech
• self-supervised pretraining
• speech representation
• downstream tasks
• efficient processing

Innovation

• Vector-quantization Learning
• Acoustic Stacking
• Receptive Field Alignment
• Mask Selection Strategy
• Self-supervised Speech Pretraining