Adaptive Modality Reliability Diagnosis and Restoration for Robust Multimodal Intent Recognition

2026-08-04

AI Summary

This work addresses the unreliability of individual modalities in multimodal intent recognition, which often arises from noise, missing data, semantic conflicts, or excessive dominance, and notes that existing methods lack mechanisms to assess modality reparability. To overcome this limitation, the paper proposes PRIME, a novel framework that introduces, for the first time, a label-free reparability diagnosis mechanism operating without explicit modality reliability annotations. PRIME employs a closed-loop pipeline to jointly diagnose, repair, and re-evaluate modality reliability at the sample level. It integrates heteroscedastic uncertainty modeling with multidimensional diagnostic signals—such as cognitive disagreement and cross-modal consistency—to drive a prototype-conditioned variational restoration module that reconstructs missing or corrupted evidence using complementary modalities. Robustness is further enhanced through inverse-variance fusion. Experiments demonstrate that PRIME achieves competitive performance on standard benchmarks and significantly outperforms state-of-the-art methods under various perturbations, including modality missingness, noise, conflict, and imbalance.

Abstract

Multimodal intent recognition combines linguistic, acoustic, and visual evidence, but individual modalities may be noisy, missing, semantically conflicting, or disproportionately dominant. Existing methods typically infer modality importance implicitly and either reweight or suppress unreliable inputs, without determining whether a degraded modality can be repaired and subsequently trusted. We propose PRIME (Precision-weighted Reliability Inference and Modality rEstoration), a closed-loop reliability guided framework that jointly diagnoses, restores, and reassesses modality quality at the sample level. PRIME represents the weakness of each modality through a contextual log-variance estimated from complementary diagnostic evidence, including predictive confidence, epistemic disagreement, cross-modal consensus, and feature degeneracy. Because modality-reliability annotations are unavailable, the estimator is explicitly trained using controlled modality corruption with known degradation severity, together with a heteroscedastic uncertainty objective. Rather than directly discarding an unreliable modality, PRIME uses its estimated weakness to control a prototype-conditioned variational restoration module that reconstructs the degraded representation from complementary modalities. Crucially, reliability is re-estimated after restoration, allowing the model to determine whether the repaired representation has become sufficiently trustworthy to contribute to prediction. The resulting post-restoration precisions are used for inverse-variance multimodal fusion. Experiments on multimodal intent-recognition benchmarks show that PRIME maintains competitive clean-data performance while improving robustness under missing, noisy, conflicting, and modality-imbalanced conditions.

Problem

• multimodal intent recognition
• modality reliability
• noisy modalities
• missing modalities
• semantic conflict

Innovation

• modality reliability diagnosis
• variational restoration
• heteroscedastic uncertainty
• inverse-variance fusion
• closed-loop multimodal learning