uDistil-Whisper: Label-Free Data Filtering for Knowledge Distillation in Low-Data Regimes

2024-07-01

AI Summary

Knowledge distillation for low-resource automatic speech recognition (ASR) is bottlenecked by its reliance on human-annotated labels. Method: This paper proposes the first zero-label Whisper knowledge distillation framework, featuring an unsupervised data quality assessment mechanism that integrates confidence self-calibration, consistency regularization, and dynamic sample weighting for pseudo-label filtering; it further employs a lightweight Whisper sub-architecture and a curriculum-based distillation paradigm. Contribution/Results: To our knowledge, this is the first work achieving efficient ASR model distillation without any ground-truth labels—eliminating dependence on annotated data for label filtering. Experiments demonstrate that the best distilled model achieves a 5–7 percentage-point lower word error rate (WER) than the Whisper teacher, with 25–50% reductions in computational and memory overhead. Moreover, it consistently outperforms both zero-shot and supervised filtering baselines across both low-data and data-augmented scenarios.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Diffusion Models for Vision

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Recent work on distilling Whisper's knowledge into small models using pseudo-labels shows promising performance while reducing the size by up to 50%. This results in small, efficient, and dedicated models. However, a critical step of distillation using pseudo-labels involves filtering high-quality predictions and using only those during training. This step requires ground truth labels to compare with and filter low-quality examples, making the process dependent on human labels. Additionally, the distillation process requires a large amount of data thereby limiting its applicability in low-resource settings. To address this, we propose a distillation framework that does not require any labeled data. Through experimentation, we show that our best-distilled models outperform the teacher model by 5-7 WER points and are on par with or outperform similar supervised data filtering setups. When scaling the data, our models significantly outperform all zero-shot and supervised models. Our models are also 25-50% more compute- and memory-efficient while maintaining performance equal to or better than that of the teacher model. For more details about our models, dataset, and other resources, please visit our GitHub page: https://github.com/UBC-NLP/uDistilWhisper.

Problem

• Label-free data filtering for knowledge distillation
• Improves efficiency in low-data regimes
• Outperforms teacher model without labeled data

Innovation

• Label-free distillation framework
• Outperforms teacher model
• Compute and memory efficient