Rethinking Modality Reliability in Multimodal Sentiment Analysis with Incomplete Observations

2026-08-04

AI Summary

This work addresses the challenge of multimodal sentiment analysis in real-world scenarios, where performance is often hindered by missing observational data and the lack of explicit modeling of modality reliability in existing methods, leading to reliability mismatch and propagation bias. To overcome these limitations, the authors propose the Modality Reliability-aware Collaborative Fusion (MRCF) framework, which introduces, for the first time, a sample-level modality reliability assessment mechanism that integrates intra-modality quality cues with cross-modality semantic consistency. MRCF dynamically regulates multimodal information flow through a reliability-aware branch, a reliability-guided interaction mechanism, and a calibration-based fusion module. Extensive experiments on CMU-MOSI, CMU-MOSEI, and CH-SIMS demonstrate that the proposed approach significantly enhances model robustness and accuracy under incomplete observational conditions.

Abstract

Multimodal Sentiment Analysis (MSA) integrates text, audio, and vision to infer human affect, yet real-world multimodal observations are often incomplete. Existing methods for incomplete-observation MSA mainly follow two paradigms. Reconstruction-based methods recover missing information from observed modalities, while joint-representation methods learn directly from incomplete inputs. Although effective, these methods usually treat modality reliability only implicitly within representation learning or fusion design rather than modeling it explicitly. We argue that modality reliability is a central variable in incomplete-observation settings. Failure to model it explicitly gives rise to two related issues. The first is reliability mismatch, in which the affective evidence retained by each modality varies across samples and missing rates. The second is reliability propagation bias, in which messages from degraded modalities may adversely affect cross-modal interaction and predictive performance. To address these issues, we propose MRCF, a Modality Reliability-Calibrated Framework for MSA with incomplete observations. MRCF contains a Reliability-Aware Branch that estimates sample-specific modality reliability from intramodal quality cues and cross-modal semantic consistency, a Reliability-Guided Interaction Branch that uses the estimated scores to modulate cross-modal information flow, and a Reliability-Calibrated Fusion Module that integrates reliability and semantic cues for final prediction. Experiments on CMU-MOSI, CMU-MOSEI, and CH-SIMS show that MRCF achieves strong performance under standard incomplete-observation protocols. Further analyses provide evidence that explicit reliability modeling helps mitigate reliability mismatch and reliability propagation bias during interaction and fusion.

Problem

• Multimodal Sentiment Analysis
• Incomplete Observations
• Modality Reliability
• Reliability Mismatch
• Reliability Propagation Bias

Innovation

• Modality Reliability
• Incomplete Observations
• Multimodal Sentiment Analysis
• Reliability-Calibrated Fusion
• Cross-modal Interaction