Uncertainty-Aware Multimodal Learning via Conformal Shapley Intervals

2026-01-30

AI Summary

This work addresses the challenge in multimodal learning that modalities often contribute unevenly and data-dependently, making it difficult to assess their effectiveness and reliability. To tackle this, the study introduces a novel framework that integrates Shapley values with conformal inference to construct statistically guaranteed confidence intervals for modality importance. Building upon this, the authors derive a theoretically optimal modality selection strategy. The proposed method not only quantifies uncertainty in modality importance but also demonstrates strong empirical performance across multiple datasets: competitive predictive accuracy is achieved using only a few critical modalities, while simultaneously providing reliable uncertainty estimates.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Reasoning under Uncertainty: Uncertainty Representations

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Security and Privacy: Data transparency and provenance
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Multimodal learning combines information from multiple data modalities to improve predictive performance. However, modalities often contribute unequally and in a data dependent way, making it unclear which data modalities are genuinely informative and to what extent their contributions can be trusted. Quantifying modality level importance together with uncertainty is therefore central to interpretable and reliable multimodal learning. We introduce conformal Shapley intervals, a framework that combines Shapley values with conformal inference to construct uncertainty-aware importance intervals for each modality. Building on these intervals, we propose a modality selection procedure with a provable optimality guarantee: conditional on the observed features, the selected subset of modalities achieves performance close to that of the optimal subset. We demonstrate the effectiveness of our approach on multiple datasets, showing that it provides meaningful uncertainty quantification and strong predictive performance while relying on only a small number of informative modalities.

Problem

• multimodal learning
• modality importance
• uncertainty quantification
• Shapley values
• conformal inference

Innovation

• Conformal Inference
• Shapley Values
• Multimodal Learning
• Uncertainty Quantification
• Modality Selection