Enhancing Speaker Verification with w2v-BERT 2.0 and Knowledge Distillation guided Structured Pruning

2025-10-05

AI Summary

To address the high parameter count and computational cost of large-scale self-supervised pre-trained models (PTMs) in speaker verification (SV), this paper proposes an efficient fine-tuning and compression co-design framework. It leverages w2v-BERT 2.0 for speech representation extraction, integrates a Multi-layer Feature Aggregation (MFA) structure with Layer Adapters to fuse hierarchical features, employs Low-Rank Adaptation (LoRA) for parameter-efficient fine-tuning, and applies knowledge distillation-guided structured pruning. The framework achieves near-lossless performance while removing 80% of parameters. On the VoxCeleb1-O and VoxCeleb1-H test sets, it attains EERs of 0.12% and 0.55%, respectively; after pruning, EER increases by only 0.04%, outperforming state-of-the-art methods significantly. This work demonstrates a principled trade-off between accuracy and efficiency in SV, enabling scalable deployment of large PTMs without compromising verification performance.

Technology Category

• Natural Language Processing: Speech
• Machine Learning: Unsupervised & Self-Supervised Learning
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Web Mining and Content Analysis: Large pretrained models with web data
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Large-scale self-supervised Pre-Trained Models (PTMs) have shown significant improvements in the speaker verification (SV) task by providing rich feature representations. In this paper, we utilize w2v-BERT 2.0, a model with approximately 600 million parameters trained on 450 million hours of unlabeled data across 143 languages, for the SV task. The MFA structure with Layer Adapter is employed to process the multi-layer feature outputs from the PTM and extract speaker embeddings. Additionally, we incorporate LoRA for efficient fine-tuning. Our model achieves state-of-the-art results with 0.12% and 0.55% EER on the Vox1-O and Vox1-H test sets, respectively. Furthermore, we apply knowledge distillation guided structured pruning, reducing the model size by 80% while achieving only a 0.04% EER degradation. Source code and models are released at https://github.com/ZXHY-82/w2v-BERT-2.0_SV.

Problem

• Improving speaker verification using large-scale pre-trained w2v-BERT 2.0 model
• Applying knowledge distillation to reduce model size by 80%
• Achieving state-of-the-art performance with minimal EER degradation

Innovation

• Uses w2v-BERT 2.0 for speaker verification
• Employs MFA structure with Layer Adapter
• Applies knowledge distillation guided structured pruning