Should Missing Modalities Always Be Necessary to Repair for Multi-modal Sentiment Analysis?

2026-07-19

AI Summary

This work challenges the prevailing assumption in multimodal sentiment analysis that all missing modalities must be reconstructed under partial observability. To address this, the authors propose SIEVE, a novel framework featuring a sample-adaptive modality reconstruction mechanism. SIEVE employs a dual-branch architecture to directly compare the losses of reconstructing versus not reconstructing missing modalities, generating an empirical sufficiency signal. By integrating evidential deep learning with cognitive uncertainty modeling, it implements an evidence-gated mechanism that dynamically decides, on a per-sample basis, whether reconstruction is necessary. Notably, SIEVE operates as a plug-and-play module without requiring modifications to underlying reconstruction components. Experiments on CMU-MOSI and IEMOCAP demonstrate consistent and significant performance gains across diverse backbone models, approaching the theoretical sample-level optimum.

Abstract

Existing methods for multimodal sentiment analysis (MSA) under missing modalities usually follow a repair-first paradigm. We revisit this assumption and ask: \emph{should every missing modality be repaired?} A per-sample oracle analysis shows the answer is not always: full-modality input is optimal for only a small fraction of samples, and every modality subset is preferred by some samples. These results suggest that adding or repairing modalities may not always improve prediction, and that the utility of each modality is sample-dependent. Building on this finding, we propose \textbf{S}ufficiency-\textbf{I}nformed \textbf{E}vidential \textbf{V}al\textbf{vE} (\textbf{SIEVE}) that turns ``whether to repair'' into an explicit, learnable decision at the sample level. SIEVE compares a direct prediction branch with a repair branch, derives an empirical sufficiency signal from their per-sample loss gap, and routes each input through an evidential gate that jointly models sufficiency and its epistemic uncertainty. SIEVE is repair-agnostic: it operates as a plug-and-play decision on top of any explicit or implicit repair module, without modifying its internal design. Experiments on CMU-MOSI and IEMOCAP show that SIEVE consistently improves representative repair backbones across evaluated missing rates, and approaches the per-sample dual-branch achievable optimum.

Problem

• multi-modal sentiment analysis
• missing modalities
• modality repair
• sample-dependent utility
• prediction sufficiency

Innovation

• modality sufficiency
• evidential reasoning
• missing modality
• sample-level decision
• repair-agnostic