🤖 AI Summary
To address insufficient cross-modal fusion and excessive parameter overhead in multimodal model fine-tuning, this paper proposes the Heterogeneous Mixture-of-Experts Adapter (Heterogeneous MoE Adapter), the first to integrate heterogeneous Mixture of Experts into parameter-efficient fine-tuning (PEFT) for multimodal models. Under frozen backbone parameters, our method introduces modality-aware low-rank affine experts and a dynamic gating mechanism to enable efficient cross-modal collaboration and deep semantic fusion. Evaluated on eight vision-audio and vision-text downstream tasks, it achieves state-of-the-art performance while tuning only 5–8% of the total parameters—significantly outperforming unimodal PEFT baselines. This demonstrates the critical role of heterogeneous expert modeling in enhancing multimodal representation learning.
📝 Abstract
Multi-modal models excel in cross-modal tasks but are computationally expensive due to their billions of parameters. Parameter-efficient fine-tuning (PEFT) offers a solution by adding small trainable components while freezing pre-trained parameters. However, existing methods primarily focus on uni-modal processing, overlooking the critical modal fusion needed for multi-modal tasks. To fill this gap, we propose heterogeneous mixture of experts adapters that extend the traditional PEFT framework to support multi-modal expert combinations and improve information interaction. Additionally, our approach modifies the affine linear expert design to enable efficient modal fusion in a low-rank space, achieving competitive performance with only 5-8% of the parameters fine-tuned. Experiments across eight downstream tasks, including visual-audio and text-visual, demonstrate the superior performance of the approach.