Fine-Tuning Whisper for Inclusive Prosodic Stress Analysis

2025-03-03

AI Summary

Automatic prosodic accent recognition—encompassing phrasal, lexical, and contrastive accents—remains challenging; moreover, current ASR systems exhibit limited inclusivity for neurodiverse individuals and speakers across gender identities. Method: We propose the first fine-grained multi-task fine-tuning framework based on Whisper-large-v2, uniquely adapting it to jointly recognize all three accent types while simultaneously classifying neurotype and gender. We construct and manually annotate a neurodiverse speech dataset and introduce bias-mitigating multi-task learning to disentangle prosodic, neurocognitive, and sociodemographic attributes. Contribution/Results: Our model achieves near-human accent recognition accuracy (≥92%), with neurotype and gender classification accuracies exceeding 99%. It significantly enhances ASR robustness and fairness across diverse articulatory patterns and speaker demographics, advancing interpretable and inclusive speech technologies.

Technology Category

• Natural Language Processing: Speech
• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: AI for Accessibility

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: Accountability, Transparency, and Ethics for personalization
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Prosody plays a crucial role in speech perception, influencing both human understanding and automatic speech recognition (ASR) systems. Despite its importance, prosodic stress remains under-studied due to the challenge of efficiently analyzing it. This study explores fine-tuning OpenAI's Whisper large-v2 ASR model to recognize phrasal, lexical, and contrastive stress in speech. Using a dataset of 66 native English speakers, including male, female, neurotypical, and neurodivergent individuals, we assess the model's ability to generalize stress patterns and classify speakers by neurotype and gender based on brief speech samples. Our results highlight near-human accuracy in ASR performance across all three stress types and near-perfect precision in classifying gender and neurotype. By improving prosody-aware ASR, this work contributes to equitable and robust transcription technologies for diverse populations.

Problem

• Fine-tuning Whisper for prosodic stress analysis.
• Improving ASR for diverse speaker populations.
• Enhancing gender and neurotype classification accuracy.

Innovation

• Fine-tuned Whisper model for prosodic stress analysis
• Dataset includes diverse neurotypes and genders
• Achieves near-human accuracy in stress recognition