modality reliability estimation

This work addresses the unclear role of reliability scores in existing quality-aware multimodal fusion methods—specifically, whether these scores genuinely guide model decisions. To investigate this, the authors propose a leakage-safe diagnostic approach: during inference, the model is frozen and reliability scores are shuffled across test samples to assess their actual impact on final predictions. This method effectively distinguishes whether the scores actively drive fusion decisions or merely correlate with performance. Experiments on the StressID and CMU-MOSEI datasets reveal that shuffling scores has negligible effect on performance in real-world scenarios; significant gains from the fusion mechanism occur only when the reliability scores accurately predict the correctness of individual modalities.

This work addresses the challenge that modality reliability in multimodal fusion often varies dynamically with context—such as sensor degradation or class-specific noise—rendering traditional methods based on static assumptions ineffective. To overcome this limitation, the authors propose the C²MF framework, which introduces a Context-Specific Information Credibility (CSIC) metric to dynamically assess the reliability of each modality at the instance level under the current input, leveraging conditional probability circuits. Fusion is then adaptively performed via KL divergence, enabling both precision and interpretability. Experimental results demonstrate that C²MF achieves up to a 29% accuracy improvement over static baselines in high-noise and modality-conflict scenarios, while preserving the inherent interpretability of probabilistic circuits.

This work addresses the tendency of multimodal large language models to exhibit overconfidence and visual hallucinations when faced with insufficient visual evidence or semantic conflicts, compounded by the absence of instance-level reliability assessment mechanisms. To enhance model calibration without requiring retraining, the authors propose a retrieval-augmented, reliability-aware reasoning framework. This approach leverages an external visual evidence bank, integrating pretrained visual embeddings, normalized nearest-neighbor retrieval, and multidimensional reliability metrics—including similarity strength, category consistency, evidence boundary, and entropy-based uncertainty—alongside a decision gating mechanism that dynamically chooses to accept, respond cautiously, or abstain from answering. Evaluated on ImageNet-100, the method improves the accuracy of accepted predictions from 85.84% to 88.88% at a coverage rate of 89.04%, while reducing hallucination error rates by 3.04 percentage points.

This work addresses the opacity of reasoning processes in multimodal large language models (MLLMs) by identifying a novel diagnostic failure mode—“modality disruption”: high-confidence unimodal errors dominate multimodal fusion decisions, thereby suppressing corroborative evidence from other modalities. To diagnose this phenomenon, we propose a lightweight, model-agnostic framework that treats each modality as an independent agent. The framework integrates candidate label generation, self-evaluation prompting, and aggregation-based fusion to enable interpretable auditing of both modality-specific contributions and disruptive behaviors. Crucially, it is the first method to systematically characterize the dynamics of multimodal conflict. Evaluated on sentiment analysis benchmarks, it effectively disentangles data bias from intrinsic model deficiencies. Our approach provides a transferable, principled diagnostic tool for assessing the reasoning reliability of MLLMs.

RGB-D multimodal fusion mechanisms have long suffered from poor interpretability, and the fundamental nature of cross-modal complementarity remains unclear. Method: This paper establishes the first interpretability analysis framework specifically targeting the fusion process, introducing a joint metric of semantic variance and feature similarity to systematically characterize cross-modal representation consistency, intra-modal evolutionary patterns, and collaborative optimization logic. Through cross-layer feature comparison and quantitative semantic analysis, we identify a prevalent imbalance between consistency and specificity in mainstream fusion strategies. Contribution/Results: We formalize a “specificity-driven inference under consistency constraints” principle that explicates cross-modal complementarity. Our framework provides both theoretical foundations and a verifiable evaluation paradigm for designing trustworthy, generalizable multimodal fusion models.

This work addresses the challenge in multimodal sentiment analysis that different modalities exhibit significantly varying reliability at the utterance level due to factors such as occlusion, noise, or transcription errors, which can adversely affect traditional fusion approaches. To mitigate this issue, the authors propose MRUF, a method that enables adaptive fusion through multi-granularity routing and uncertainty-aware calibration. Specifically, MRUF incorporates modality importance supervision derived from leave-one-out error increases, an inverse-variance reweighting mechanism for uncertainty-based gating, and a modality-invariant contrastive alignment strategy. Experimental results demonstrate that MRUF consistently outperforms strong baselines on both aligned and unaligned settings of the CMU-MOSI and CMU-MOSEI datasets, effectively down-weighting contributions from high-uncertainty modalities and thereby enhancing model robustness.

This work addresses the challenge in multimodal fusion where unknown modality reliability can lead to over-weighting of unreliable or redundant modalities, thereby degrading predictive performance. To mitigate this, the authors propose TIER-MoE, a novel mixture-of-experts architecture that, for the first time, jointly incorporates sample-level conditional modality risk—estimated via leave-one-out prediction loss—and expert subspace compatibility to enable risk-aware sparse modality-to-expert routing. The model retains a shared multimodal pathway to preserve complementary information across modalities. Evaluated on four biomedical multimodal datasets, TIER-MoE consistently outperforms state-of-the-art methods, achieving significant improvements in both Macro-F1 and Brier score, while also demonstrating strong zero-shot external generalization capabilities.

Existing multimodal fusion models often lack robustness under noisy or uninformative data and fail to dynamically assess data quality or produce reliable confidence estimates, limiting their applicability in high-stakes clinical settings. To address these challenges, this work proposes the Adaptive Confidence-weighted Extension (ACE) framework, which uniquely integrates intra-modality correlation–driven complementary modality generation with a dual-level dynamic confidence mechanism. This enables adaptive weighting of modality reliability and outputs a global trust score. Evaluated on four multi-omics datasets—BRCA, KIPAN, LGG, and ROSMAP—ACE significantly outperforms current methods, achieving notable improvements in both classification accuracy and confidence calibration, thereby enhancing model robustness and clinical trustworthiness.

Current vision-language models (VLMs) suffer from “functional blindness” in multimodal reasoning, overly relying on linguistic priors while neglecting visual inputs, thereby undermining their reliability. This work proposes an information-theoretic modality translation protocol that quantifies a model’s genuine use of visual information through semantic payload translation rather than data ablation. We introduce the novel concept of “visual toll” and define three new metrics—Toll, Curse, and Fallacy—alongside a Semantic Sufficiency Criterion (SSC). Our analysis reveals a divergence law in multimodal scaling, demonstrating that the visual bottleneck in mainstream VLMs intensifies as language model capacity grows. These findings provide both a theoretical foundation and a design blueprint for building trustworthy multimodal systems.

This study addresses the challenge of missing modalities in clinical multimodal survival prediction, where existing methods often lack robustness. The authors propose the EMMS model, which innovatively treats missing modalities as vacuous evidence and integrates Dempster–Shafer evidence theory with Gaussian random fuzzy numbers. Without imputing missing data, EMMS jointly models epistemic and aleatoric uncertainties and dynamically evaluates modality reliability. Evaluated on four cancer datasets, the method achieves state-of-the-art performance while delivering computationally efficient, well-calibrated, and interpretable uncertainty estimates.