Improving Noise Robust Audio-Visual Speech Recognition via Router-Gated Cross-Modal Feature Fusion

2025-08-26

AI Summary

To address the challenge of dynamically assessing audio reliability in audio-visual speech recognition (AVSR) systems under noisy conditions—leading to modality dependency mismatch—this paper proposes a routing-gated cross-modal fusion framework. The method introduces a token-level acoustic degradation scoring mechanism, integrated with a cross-modal feature fusion router and gated cross-attention, enabling real-time perception of audio reliability and adaptive modulation of modality weights at each decoder layer. Evaluated on the LRS3 dataset, our approach achieves relative word error rate reductions of 16.51%–42.67% over AV-HuBERT, significantly enhancing robustness in realistic noisy scenarios. The core contribution lies in the first incorporation of audio reliability modeling into token-level dynamic weighting and gated cross-attention, thereby enabling fine-grained, interpretable cross-modal collaborative decision-making.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Multi-modal Vision

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Security and Privacy: Large-scale security measurements

Abstract

Robust audio-visual speech recognition (AVSR) in noisy environments remains challenging, as existing systems struggle to estimate audio reliability and dynamically adjust modality reliance. We propose router-gated cross-modal feature fusion, a novel AVSR framework that adaptively reweights audio and visual features based on token-level acoustic corruption scores. Using an audio-visual feature fusion-based router, our method down-weights unreliable audio tokens and reinforces visual cues through gated cross-attention in each decoder layer. This enables the model to pivot toward the visual modality when audio quality deteriorates. Experiments on LRS3 demonstrate that our approach achieves an 16.51-42.67% relative reduction in word error rate compared to AV-HuBERT. Ablation studies confirm that both the router and gating mechanism contribute to improved robustness under real-world acoustic noise.

Problem

• Enhancing noise-robust audio-visual speech recognition
• Dynamically adjusting modality reliance in noisy environments
• Adaptively reweighting features based on acoustic corruption

Innovation

• Router-gated cross-modal feature fusion
• Token-level acoustic corruption scoring
• Gated cross-attention reinforcement mechanism