Leveraging Allophony in Self-Supervised Speech Models for Atypical Pronunciation Assessment

2025-02-10

AI Summary

This study addresses the challenge of modeling allophonic variation in atypical pronunciation assessment. We propose a novel method that explicitly models phoneme-specific multi-subcluster distributions—departing from conventional single-phoneme classification paradigms. Our approach uniquely integrates frozen self-supervised speech model (S3M) features with Gaussian mixture models (GMMs) to perform environment-dependent allophone subclustering and probabilistic modeling. Evaluated on five standard benchmarks, the method achieves state-of-the-art performance on four datasets, significantly improving discrimination of articulation disorders and second-language pronunciation errors. Moreover, S3M features demonstrate superior discriminability over MFCCs and Mel-spectrograms for capturing allophonic acoustic distinctions. The core contribution is the first dedicated evaluation framework for explicit allophonic modeling, delivering an interpretable and robust acoustic representation scheme for atypical pronunciation analysis.

Technology Category

• Natural Language Processing: Speech
• Machine Learning: Clustering
• Search and Optimization: Mixed Discrete/Continuous Search

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs

Abstract

Allophony refers to the variation in the phonetic realization of a phoneme based on its phonetic environment. Modeling allophones is crucial for atypical pronunciation assessment, which involves distinguishing atypical from typical pronunciations. However, recent phoneme classifier-based approaches often simplify this by treating various realizations as a single phoneme, bypassing the complexity of modeling allophonic variation. Motivated by the acoustic modeling capabilities of frozen self-supervised speech model (S3M) features, we propose MixGoP, a novel approach that leverages Gaussian mixture models to model phoneme distributions with multiple subclusters. Our experiments show that MixGoP achieves state-of-the-art performance across four out of five datasets, including dysarthric and non-native speech. Our analysis further suggests that S3M features capture allophonic variation more effectively than MFCCs and Mel spectrograms, highlighting the benefits of integrating MixGoP with S3M features.

Problem

• Assess atypical pronunciation using allophony.
• Model phoneme distributions with multiple subclusters.
• Improve speech assessment for dysarthric and non-native speakers.

Innovation

• MixGoP uses Gaussian mixture models
• MixGoP integrates with self-supervised speech models
• S3M features capture allophonic variation effectively