Cohort-Based Active Modality Acquisition

2025-05-22

AI Summary

To address the practical challenge of missing modalities during inference in multimodal learning—where reacquisition is costly—we propose a novel cohort-level active modality acquisition paradigm. We formally define the test-time cohort-level active modality acquisition problem for the first time. Our method jointly leverages generative modality imputation and discriminative uncertainty modeling to estimate the expected performance gain from acquiring each missing modality per sample. Furthermore, we design an interpretable, upper-bound-guided heuristic scoring mechanism to prioritize acquisitions. Extensive experiments on multiple standard multimodal benchmarks demonstrate that our approach significantly outperforms unimodal baselines, entropy-guided strategies, and random selection under limited acquisition budgets—yielding an average 12.7% improvement in overall prediction performance. The framework achieves strong effectiveness, interpretability, and practical applicability.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Constraint Satisfaction and Optimization: Constraint Learning and Acquisition

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Real-world machine learning applications often involve data from multiple modalities that must be integrated effectively to make robust predictions. However, in many practical settings, not all modalities are available for every sample, and acquiring additional modalities can be costly. This raises the question: which samples should be prioritized for additional modality acquisition when resources are limited? While prior work has explored individual-level acquisition strategies and training-time active learning paradigms, test-time and cohort-based acquisition remain underexplored despite their importance in many real-world settings. We introduce Cohort-based Active Modality Acquisition (CAMA), a novel test-time setting to formalize the challenge of selecting which samples should receive additional modalities. We derive acquisition strategies that leverage a combination of generative imputation and discriminative modeling to estimate the expected benefit of acquiring missing modalities based on common evaluation metrics. We also introduce upper-bound heuristics that provide performance ceilings to benchmark acquisition strategies. Experiments on common multimodal datasets demonstrate that our proposed imputation-based strategies can more effectively guide the acquisition of new samples in comparison to those relying solely on unimodal information, entropy guidance, and random selections. Our work provides an effective solution for optimizing modality acquisition at the cohort level, enabling better utilization of resources in constrained settings.

Problem

• Prioritize samples for costly modality acquisition
• Optimize test-time cohort-based modality selection
• Improve multimodal predictions with limited resources

Innovation

• Cohort-based Active Modality Acquisition (CAMA) strategy
• Generative imputation and discriminative modeling combination
• Upper-bound heuristics for performance benchmarking