Multi-View Speech Representation Learning for Parkinson's Disease Detection Using Context-guided Cross-modal Attention

2026-06-08

AI Summary

This study addresses the limitations of existing Parkinson’s disease (PD) speech detection methods, which often rely on single-feature representations and thus fail to comprehensively capture pathological cues across heterogeneous feature spaces. To overcome this, the authors propose a multi-branch deep learning framework that processes Log-Mel spectrograms using ResNet-18, models MFCC sequences with a bidirectional LSTM, and encodes raw audio via a pretrained HuBERT model. A novel context-guided cross-modal attention mechanism is introduced to dynamically fuse these diverse speech representations. Evaluated on the PC-GITA corpus, the proposed method achieves an accuracy of 91.51%, an F1-score of 91.24%, and an AUC of 95.97%, significantly outperforming current state-of-the-art approaches and demonstrating enhanced sensitivity to PD-related vocal abnormalities.

Abstract

Parkinson's disease (PD) is a progressive neurodegenerative disorder that frequently causes speech impairments associated with hypokinetic dysarthria. As speech production relies on the precise coordination of complex neuromuscular mechanisms, speech analysis has emerged as a promising non-invasive and cost-effective biomarker for early PD detection. Recent deep learning approaches have shown encouraging results; however, most existing methods rely on a single speech representation, potentially overlooking complementary pathological information encoded across different feature spaces. In this work, we propose a multi-branch deep learning framework for automatic PD detection from speech. Each recording is segmented into 5-second chunks and represented using three complementary modalities: Log-Mel spectrograms, MFCCs, and HuBERT embeddings extracted from raw waveforms. The spectrograms are processed using a pre-trained ResNet-18 encoder, MFCC sequences are modeled through a BiLSTM network, and raw speech is encoded using a pre-trained HuBERT model. To effectively integrate these heterogeneous representations, we introduce a context-guided cross-modal attention mechanism that dynamically weights temporal HuBERT embeddings according to the global acoustic context derived from the spectrogram and MFCC branches. Experiments conducted on the publicly available Spanish PC-GITA corpus under strict speaker-independent 5-fold cross-validation demonstrate the effectiveness of the proposed approach. The proposed architecture achieves an accuracy of 91.51%, an F1-score of 91.24%, and an AUC of 95.97%. Furthermore, ablation studies confirm the contribution of both the proposed context-guided cross-modal attention mechanism and the integration of complementary speech representations. These findings highlight the potential of heterogeneous speech modeling for robust and clinically reliable PD detection.

Problem

• Parkinson's disease detection
• speech representation learning
• multi-view learning
• cross-modal attention
• hypokinetic dysarthria

Innovation

• multi-view representation
• cross-modal attention
• Parkinson's disease detection
• speech biomarker
• context-guided fusion