modality confidence estimation

This work addresses the unclear role of reliability scores in existing quality-aware multimodal fusion methods—specifically, whether these scores genuinely guide model decisions. To investigate this, the authors propose a leakage-safe diagnostic approach: during inference, the model is frozen and reliability scores are shuffled across test samples to assess their actual impact on final predictions. This method effectively distinguishes whether the scores actively drive fusion decisions or merely correlate with performance. Experiments on the StressID and CMU-MOSEI datasets reveal that shuffling scores has negligible effect on performance in real-world scenarios; significant gains from the fusion mechanism occur only when the reliability scores accurately predict the correctness of individual modalities.

In multimodal recommendation, unreliable modality data often degrades fusion performance below unimodal baselines. Existing weighted fusion methods lack effective supervision for modality reliability, leading to inaccurate weight learning. To address this, we propose a reliability-guided dynamic multimodal fusion framework. First, we implicitly formulate the Bayesian Personalized Ranking (BPR) objective as a proxy label for modality reliability and design a confidence-aware mechanism to adaptively calibrate supervision strength, thereby mitigating erroneous supervision. Second, we introduce a modality-specific score discrepancy modeling module and an end-to-end differentiable fusion module. Extensive experiments on three real-world datasets demonstrate significant improvements over state-of-the-art methods, validating the effectiveness of our reliability supervision mechanism in enhancing both fusion accuracy and robustness.

为解决微服务系统中多模态异常检测的跨模态信息利用问题，提出了一种基于置信度引导的跨模态知识转移方法（CMT-AD），通过估计模态可靠性并引入门控中间模态来缓解异质性。

This work addresses the challenge of multimodal sentiment analysis in real-world scenarios, where performance is often hindered by missing observational data and the lack of explicit modeling of modality reliability in existing methods, leading to reliability mismatch and propagation bias. To overcome these limitations, the authors propose the Modality Reliability-aware Collaborative Fusion (MRCF) framework, which introduces, for the first time, a sample-level modality reliability assessment mechanism that integrates intra-modality quality cues with cross-modality semantic consistency. MRCF dynamically regulates multimodal information flow through a reliability-aware branch, a reliability-guided interaction mechanism, and a calibration-based fusion module. Extensive experiments on CMU-MOSI, CMU-MOSEI, and CH-SIMS demonstrate that the proposed approach significantly enhances model robustness and accuracy under incomplete observational conditions.

本文通过定义参数、基于游程的非参数和数据驱动测试来评估数据间隔中的模式特征，以确定多模态的存在和位置，并在两种情况下检查这些测试的有效性。

Multimodal clinical AI systems often suffer performance degradation in deployment due to missing modalities, yet it remains challenging to identify which modality fails and whether its failure is detectable (“loud”) or undetectable (“silent”). This work proposes a model-agnostic, fine-grained failure analysis framework that relies solely on observable signals during deployment. By leveraging modality embeddings, mask-aware probes, and labels, the method enables per-sample, per-modality failure attribution and outputs both a modality complementarity matrix and loud/silent failure profiles. In simulated data, the framework accurately recovers predefined modality dominance relationships; in the real-world MIMIC-IV cohort, it reveals that missing echocardiography nearly doubles error rates and, for the first time, quantifies the proportions of monitorable versus unmonitorable failures across modalities.

This work addresses the unreliability of individual modalities in multimodal intent recognition, which often arises from noise, missing data, semantic conflicts, or excessive dominance, and notes that existing methods lack mechanisms to assess modality reparability. To overcome this limitation, the paper proposes PRIME, a novel framework that introduces, for the first time, a label-free reparability diagnosis mechanism operating without explicit modality reliability annotations. PRIME employs a closed-loop pipeline to jointly diagnose, repair, and re-evaluate modality reliability at the sample level. It integrates heteroscedastic uncertainty modeling with multidimensional diagnostic signals—such as cognitive disagreement and cross-modal consistency—to drive a prototype-conditioned variational restoration module that reconstructs missing or corrupted evidence using complementary modalities. Robustness is further enhanced through inverse-variance fusion. Experiments demonstrate that PRIME achieves competitive performance on standard benchmarks and significantly outperforms state-of-the-art methods under various perturbations, including modality missingness, noise, conflict, and imbalance.

This work addresses the challenge of calibrating prediction intervals in multimodal regression, where missing modalities or inconsistent predictions often undermine reliability. The authors propose a modality-aware conformal calibration layer that equips each modality with an independent predictor and constructs a disagreement score based on their predictive discrepancies. By integrating split conformal prediction with Mondrian stratified calibration under a strict data-splitting protocol, the method adaptively adjusts prediction intervals according to the observed modality pattern at test time. Evaluated across four datasets in 60 experiments, the approach achieves CRPS performance superior or comparable to baselines in 59 cases and yields narrower intervals in 52, while consistently maintaining coverage near 95%. Notably, under modality-missing conditions, it recovers coverage by up to 19.5 percentage points.

This work investigates whether user-specific modality weighting mechanisms widely adopted in multimodal recommendation systems genuinely capture individual user preferences. To this end, the authors propose an auditing framework comprising two metrics—real-GM and real-shuf—that evaluate personalization efficacy by comparing six personalized weighting methods against global weights and shuffled user-weight assignments, all under a unified collaborative filtering backbone. Experimental results across three short-video and one cross-domain e-commerce dataset reveal that performance gains from most methods stem primarily from increased model capacity rather than authentic user signals, with gating mechanisms often inducing spurious personalization due to their reliance on shared embeddings. Notably, global modality weights already achieve nearly all attainable gains, while personalized weighting shows no consistent improvement; the proposed audit framework effectively identifies architectures that truly encode user-specific patterns.

This study addresses the confidence calibration bias caused by missing modalities in multimodal brain tumor segmentation by proposing the MMA-LTS method. Challenging the conventional assumption that task difficulty depends solely on the number of missing modalities, this work reveals that predictive uncertainty is instead determined by the specific combination of absent modalities. Accordingly, MMA-LTS introduces learnable modality-availability tokens, voxel-wise difficulty scores, and a local temperature scaling mechanism to achieve spatially adaptive, voxel-level post-hoc calibration. Experiments on the BraTS and FeTS datasets demonstrate that MMA-LTS significantly improves calibration accuracy while maintaining state-of-the-art segmentation performance, thereby effectively enhancing the clinical trustworthiness of the model.