Normal-Anchored First-Order Model-Agnostic Meta-Learning based Whisper Fine-Tuning for Enhancing Fairness of Cleft Lip and Palate Speech Recognition

2026-07-31

AI Summary

Pretrained automatic speech recognition models exhibit limited generalization on cleft lip and palate (CLP) pathological speech, particularly under low-resource conditions with heterogeneous severity levels. To address this, this work proposes NA-FOMAML, a novel approach that integrates normal speech as a stabilizing anchor in the inner loop of a first-order meta-learning framework, while grouping CLP samples by severity for outer-loop optimization of the Whisper model. This design enhances robustness and fairness toward pathological speech. Combined with frozen/full encoder fine-tuning strategies and phoneme-level analysis, the method significantly narrows the performance gap between normal and pathological speech on the NMCPC and AIISH datasets. On NMCPC, word error rates (WERs) for normal, mild, and moderate CLP reach 4.40%, 5.53%, and 16.14%, respectively, though severe CLP remains challenging.

Abstract

Automatic speech recognition (ASR) for cleft lip and palate (CLP) speech is difficult because acoustic and articulatory patterns vary across severity levels. This variability reduces the performance of pretrained ASR systems, and conventional fine-tuning may not generalize well under low-resource, heterogeneous CLP conditions. This work proposes Normal-Anchored First-Order Model-Agnostic Meta-Learning (NA-FOMAML) for adapting Whisper to CLP speech. The method uses a first-order bilevel meta-learning framework in which normal speech is used in the inner loop as a stable support condition, while CLP severity groups are used in the outer loop to improve post-adaptation robustness. This design aims to reduce the performance gap between normal and pathological speech. Experiments are conducted on the NMCPC and AIISH datasets using four normal-anchored training configurations. Frozen encoder, full encoder, and selected Whisper encoder-layer tuning strategies are evaluated, including layers 0--5, 4--11, 6--11, and 8--11, with decoder and projection-head adaptation. Results show that outer-loop training with only normal speech is insufficient. For NMCPC, full encoder tuning with Normal to Normal+Mild+Moderate gives WERs of 4.40%, 5.53%, 16.14%, and 52.07% for normal, mild, moderate, and severe speech. For AIISH, full encoder tuning with Normal to Normal+Mild+Moderate+Severe gives WERs of 2.48%, 19.66%, 14.05%, and 57.50%. A transcription-based phoneme-category analysis shows that severe CLP speech has high error rates across fricatives, affricates, nasals, liquids, plosives, and vowels. Overall, NA-FOMAML improves cross-severity robustness, but severe speech still requires severity-aware sampling, phoneme-aware loss functions, and augmentation targeting pressure consonant and resonance-related distortions.

Problem

• cleft lip and palate speech
• automatic speech recognition
• fairness
• low-resource
• severity variability

Innovation

• meta-learning
• Whisper fine-tuning
• cleft lip and palate speech
• fairness in ASR
• normal-anchored adaptation