Contrastive Training with LLM-generated Near-Misses for Robust Code-Switching Speech Recognition

2026-06-05

AI Summary

This work addresses the challenge of degraded automatic speech recognition (ASR) accuracy in code-switching scenarios, particularly around critical switching regions. To this end, the authors propose a contrastive training framework that focuses on points of interest (POIs) where language switches occur. Their approach uniquely leverages a large language model to generate acoustically plausible yet confusable near-miss error samples, which are then filtered into hard negatives using multi-dimensional constraints spanning acoustic, phonetic, and textual modalities. The framework further incorporates POI-weighted cross-entropy loss and multi-negative contrastive ranking loss to fine-tune Whisper-small via LoRA adaptation. Evaluated on the CS-FLEURS (cmn-eng) and ViMedCSS (vie-eng) datasets, the method reduces both overall and code-switching-aware error rates by over 2% compared to standard LoRA fine-tuning, substantially enhancing model robustness at language transition boundaries.

Abstract

Code-switching (CS), the alternation between multiple languages within a single utterance, remains challenging for Automatic Speech Recognition (ASR). To address this issue, we propose a Point-of-Interest (POI)-aware contrastive training framework that improves recognition at CS-critical regions. We first identify CS spans by adopting POI detection method from literature, then construct acoustically plausible near-miss hypotheses by perturbing POIs in ASR N-best outputs and expanding candidates with a large language model. Hard but plausible negatives are retained through filtering with acoustic, phonemic, and textual constraints. Finally, we fine-tune Whisper-small with LoRA using a POI-weighted cross-entropy anchor objective together with a multi-negative contrastive ranking loss. Experiments on CS-FLEURS (cmn-eng) and ViMedCSS (vie-eng) show consistent reductions of over 2% in both general and CS-aware error rates compared to standard LoRA fine-tuning.

Problem

• code-switching
• automatic speech recognition
• near-miss hypotheses
• contrastive training
• CS-aware error

Innovation

• contrastive training
• code-switching
• near-miss generation
• POI-aware
• LLM-augmented ASR