Fine-tuning Whisper on Low-Resource Languages for Real-World Applications

2024-12-20

arXiv.org

AI Summary

To address Whisper’s poor adaptability to low-resource languages—caused by the scarcity of long-duration audio annotations—this paper proposes a novel method for constructing high-quality long-text corpora using only sentence-level text–speech pairs. Our approach comprises rule-guided data recombination, temporal alignment augmentation, and task-adaptive fine-tuning, all without requiring external long-audio annotations. Crucially, it preserves Whisper’s native segmentation capability while enhancing its modeling of extended audio sequences. This is the first method enabling lossless conversion from sentence-level data to high-fidelity long-text corpora, effectively circumventing copyright restrictions and annotation bottlenecks. Evaluated on Swiss German, our method achieves a new state-of-the-art (SOTA), with BLEU scores significantly surpassing both the baseline Whisper and the best-performing local models. Moreover, it demonstrates strong cross-lingual transfer potential. We publicly release training scripts and comprehensive adaptation guidelines to facilitate reproducibility and community adoption.

Technology Category

• Natural Language Processing: Speech
• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

This paper presents a new approach to fine-tuning OpenAI's Whisper model for low-resource languages by introducing a novel data generation method that converts sentence-level data into a long-form corpus, using Swiss German as a case study. Non-sentence-level data, which could improve the performance of long-form audio, is difficult to obtain and often restricted by copyright laws. Our method bridges this gap by transforming more accessible sentence-level data into a format that preserves the model's ability to handle long-form audio and perform segmentation without requiring non-sentence-level data. Our data generation process improves performance in several real-world applications and leads to the development of a new state-of-the-art speech-to-text (STT) model for Swiss German. We compare our model with a non-fine-tuned Whisper and our previous state-of-the-art Swiss German STT models, where our new model achieves higher BLEU scores. Our results also indicate that the proposed method is adaptable to other low-resource languages, supported by written guidance and code that allows the creation of fine-tuned Whisper models, which keep segmentation capabilities and allow the transcription of longer audio files using only sentence-level data with high quality.

Problem

• Fine-tuning Whisper for low-resource languages like Swiss German
• Converting sentence-level data to long-form corpus for better performance
• Overcoming copyright restrictions on non-sentence-level data usage

Innovation

• Fine-tuning Whisper for low-resource languages
• Converting sentence-level data into long-form corpus
• Maintaining segmentation without non-sentence-level data