Overcoming Decoder Inconsistencies in Whisper for Dravidian and Low-Resource Languages

2026-06-08

AI Summary

This study addresses the notably high word error rates (WER) of multilingual speech recognition models like Whisper on low-resource agglutinative languages such as Dravidian languages, primarily caused by lexical sparsity, morphological complexity, and an imbalance between self-attention and cross-attention in the decoder, which leads to character-level substitution errors. The work is the first to identify this issue and proposes two synthetic-data-free decoder optimization strategies: an adaptive weighting mechanism to balance the two attention types (Weighted-Attention) and an intermediate prediction re-injection technique to enhance decoding consistency (Self-Conditioning). Experimental results demonstrate that both methods significantly reduce WER across multiple low-resource agglutinative languages, confirming their effectiveness and generalizability.

Abstract

Multilingual ASR models such as Whisper perform well on high-resource languages but exhibit substantially higher Word Error Rates (WER) for Dravidian languages compared to Indo-Aryan ones. Through linguistic and dataset analysis, we show that Dravidian languages have longer words, higher vocabulary diversity, and lower repetition, resulting in sparse token distributions and frequent character-level substitution errors. Baseline fine-tuning further reveals decoder imbalance between self-attention (linguistic context) and cross-attention (acoustic cues). Although synthetic token-repetition experiments indicate potential gains, they are impractical. Motivated by these observations, we introduce two decoder-level enhancements: Weighted-Attention, which adaptively balances attention sources, and Self-Conditioning, which reinjects intermediate predictions to improve token consistency. Experiments demonstrate consistent WER reductions for low-resource and agglutinative languages.

Problem

• Whisper
• Dravidian languages
• low-resource languages
• decoder inconsistencies
• Word Error Rate

Innovation

• Weighted-Attention
• Self-Conditioning
• low-resource languages
• decoder imbalance
• agglutinative languages