🤖 AI Summary
Multimodal mixture-of-experts (MoE) models suffer from performance degradation when incorporating noise-dominant auxiliary modalities; existing approaches—such as partial information decomposition—struggle to scale beyond two modalities and lack instance-level dynamic modulation. This paper proposes a parameter-free, two-tier weight adaptation framework: an upper tier estimates modality-level importance via inter-modal mutual information, while a lower tier enables fine-grained, instance-specific fusion through KL-divergence-based weighting. To our knowledge, this is the first method enabling non-parametric, dynamic fusion across arbitrarily many modalities without architectural modifications or additional parameters. Evaluated on sentiment regression and clinical classification tasks, it achieves significant improvements—12.3% reduction in mean absolute error (MAE) for regression and 3.8–5.1% gains in multiclass accuracy—demonstrating strong scalability, robustness to modality noise, and cross-task generalization.
📝 Abstract
Mixture-of-Experts (MoE) models have become increasingly powerful in multimodal learning by enabling modular specialization across modalities. However, their effectiveness remains unclear when additional modalities introduce more noise than complementary information. Existing approaches, such as the Partial Information Decomposition, struggle to scale beyond two modalities and lack the resolution needed for instance-level control. We propose Beyond Two-modality Weighting (BTW), a bi-level, non-parametric weighting framework that combines instance-level Kullback-Leibler (KL) divergence and modality-level mutual information (MI) to dynamically adjust modality importance during training. Our method does not require additional parameters and can be applied to an arbitrary number of modalities. Specifically, BTW computes per-example KL weights by measuring the divergence between each unimodal and the current multimodal prediction, and modality-wide MI weights by estimating global alignment between unimodal and multimodal outputs. Extensive experiments on sentiment regression and clinical classification demonstrate that our method significantly improves regression performance and multiclass classification accuracy.