🤖 AI Summary
To address the challenge of sparse multimodal data fusion under modality missing scenarios, this paper proposes Modal Channel Attention (MCA): it explicitly models fusion embeddings for all possible modality combinations and incorporates an attention masking mechanism to enable flexible, robust, and unified fusion. Furthermore, MCA introduces multimodal embedding space alignment and contrastive fusion representation learning to ensure spatial consistency and uniformity between unimodal and fused representations. Evaluated on CMU-MOSEI (sentiment analysis) and TCGA (cancer prognosis), MCA consistently outperforms state-of-the-art baselines Zorro and EAO across classification/regression accuracy, ranking, and recall metrics. These results empirically validate the critical importance of exhaustive combinatorial contrastive fusion for modeling incomplete multimodal data.
📝 Abstract
Multimodal data fusion is essential for applications requiring the integration of diverse data sources, especially in the presence of incomplete or sparsely available modalities. This paper presents a comparative study of three multimodal embedding techniques, Modal Channel Attention (MCA), Zorro, and Everything at Once (EAO), to evaluate their performance on sparsely multimodal data. MCA introduces fusion embeddings for all combinations of input modalities and uses attention masking to create distinct attention channels, enabling flexible and efficient data fusion. Experiments on two datasets with four modalities each, CMU-MOSEI and TCGA, demonstrate that MCA outperforms Zorro across ranking, recall, regression, and classification tasks and outperforms EAO across regression and classification tasks. MCA achieves superior performance by maintaining robust uniformity across unimodal and fusion embeddings. While EAO performs best in ranking metrics due to its approach of forming fusion embeddings post-inference, it underperforms in downstream tasks requiring multimodal interactions. These results highlight the importance of contrasting all modality combinations in constructing embedding spaces and offers insights into the design of multimodal architectures for real-world applications with incomplete data.