DAIF: A Data-Driven Intermediate Fusion Framework for Multimodal Supervised Learning via Approximate Message Passing

2026-08-03

AI Summary

This study addresses the challenge of adaptively determining fusion granularity in multimodal supervised learning by proposing a data-driven intermediate fusion framework. Built upon a Bayesian multimodal factor model, the method uniquely integrates random matrix theory with nonparametric dependence measures to automatically infer inter-modal dependency structures and cluster modalities directly from data. Within each cluster, empirical Bayes priors are introduced, and an approximate message passing (AMP) algorithm is embedded to construct denoisers that yield low-dimensional features capturing both shared information and modality-specific signals. Experiments on simulated data as well as real-world TEA-seq and TCGA-BRCA datasets demonstrate that the proposed approach achieves competitive or superior performance over existing methods in tasks such as T-cell protein expression prediction and patient survival prediction.

Abstract

Multimodal supervised learning seeks to leverage multiple heterogeneous data sources to improve predictive performance. A central challenge is determining the fusion granularity across modalities: over-integration may amplify noise while under-integration fails to exploit cross-modal dependence. Existing approaches rely on pre-specified fusion architectures, from early to late fusion, that may not adapt to the underlying dependence structure among modalities. We propose DAIF, a data adaptive intermediate fusion framework that combines random matrix theory and non-parametric dependence measures to learn fusion structure directly from data. We operate under a Bayesian multimodal factor model where the prior on the latent factors determines the cross-modal dependence. Our method clusters modalities based on estimated intermodal dependence, then performs clusterwise empirical Bayes estimation of the priors. These estimated priors are used to construct denoisers within an approximate message passing (AMP) framework, yielding denoised low-dimensional features that borrow strength across related modalities while preserving modality-specific signal. The resulting embeddings are used for downstream supervised prediction. We evaluate the framework through simulations under varying dependence structures and signal regimes, comparing against several benchmark methods, and demonstrate its practical utility on two multimodal datasets, namely a trimodal TEA-seq dataset (Swanson et al., 2021) and TCGA-BRCA dataset (Goldman et al., 2020). In the first example, we predict the expression level of a T-cell differentiation marker protein and in the second case we analyze patient survival prediction based on multimodal information. Our method competes with or outperforms the state-of-the-art techniques in both prediction problems, demonstrating its versatility across diverse supervised learning tasks.

Problem

• multimodal supervised learning
• fusion granularity
• cross-modal dependence
• data-driven fusion
• heterogeneous data

Innovation

• intermediate fusion
• approximate message passing
• data-driven adaptation
• multimodal learning
• empirical Bayes