🤖 AI Summary
In multimodal learning, modality imbalance often stems from internal optimization bias within neural networks—not merely from inherent representational disparities across modalities. Existing approaches suppress dominant modalities to boost underperforming ones, inadvertently degrading overall performance. To address this, we propose Adaptive Network-Intrinsic Modulation (ANIM), the first method to decouple suboptimally trained parameters in dominant modalities and introduce lightweight auxiliary modules. ANIM jointly analyzes parameter-level optimization states and cross-layer modality imbalance, dynamically adjusting modulation strength across network depths to enable synergistic optimization of both dominant and underperforming modalities. Importantly, ANIM is architecture-agnostic—requiring no modifications to backbone networks, fusion strategies, or optimizers—ensuring broad applicability. Extensive experiments on multiple benchmarks demonstrate significant improvements over state-of-the-art methods, achieving superior modality balance while simultaneously enhancing overall model performance.
📝 Abstract
Multimodal learning has significantly enhanced machine learning performance but still faces numerous challenges and limitations. Imbalanced multimodal learning is one of the problems extensively studied in recent works and is typically mitigated by modulating the learning of each modality. However, we find that these methods typically hinder the dominant modality's learning to promote weaker modalities, which affects overall multimodal performance. We analyze the cause of this issue and highlight a commonly overlooked problem: optimization bias within networks. To address this, we propose Adaptive Intra-Network Modulation (AIM) to improve balanced modality learning. AIM accounts for differences in optimization state across parameters and depths within the network during modulation, achieving balanced multimodal learning without hindering either dominant or weak modalities for the first time. Specifically, AIM decouples the dominant modality's under-optimized parameters into Auxiliary Blocks and encourages reliance on these performance-degraded blocks for joint training with weaker modalities. This approach effectively prevents suppression of weaker modalities while enabling targeted optimization of under-optimized parameters to improve the dominant modality. Additionally, AIM assesses modality imbalance level across network depths and adaptively adjusts modulation strength at each depth. Experimental results demonstrate that AIM outperforms state-of-the-art imbalanced modality learning methods across multiple benchmarks and exhibits strong generalizability across different backbones, fusion strategies, and optimizers.