Harnessing Domain Specialists in Multimodal Mixture-of-Experts for Efficient Adaptation

2026-10-01

AI Summary

This study addresses the substantial computational overhead and inefficiency encountered when adapting multimodal Mixture-of-Experts (MoE) models to specific domains. To this end, it proposes ExpertLens, a data-free framework that exploits the semantic modularity inherent in MoE sparsity. By decoding pretrained router weights, the method precisely identifies domain-relevant experts and applies selective fine-tuning exclusively to their associated parameters. This work highlights the semantic modularity value of sparse architectures. Experimental results demonstrate that updating merely 21.7%–47.0% of the parameters achieves performance on par with full fine-tuning while accelerating training up to fourfold, significantly outperforming mainstream baselines such as LoRA.

Abstract

Mixture-of-Experts (MoE) architectures scale model capacity through sparse computation, routing each token through only a small subset of experts. In this work, we explore whether this sparsity gives rise to emergent intrinsic organization in multimodal MoEs. We find that experts develop strong semantic specialization across modalities and domains despite not being explicitly trained for modularity. Building on this structure, we introduce ExpertLens, a data-free method that identifies domain-specialized experts directly from pretrained model weights by decoding router weights into semantically meaningful vocabulary tokens. We leverage this specialization for efficient multimodal adaptation by selectively fine-tuning experts relevant to a target domain. Across math, medical, and remote sensing tasks, ExpertLens matches or surpasses full fine-tuning while updating only 21.7 - 47.0% of model parameters and achieving a 4.0x average training speedup, and outperforms LoRA in both adaptation performance and training efficiency. These results show that sparsity introduced for efficiency can give rise to semantic modularity that is directly useful for efficient adaptation.

Problem

• Multimodal Mixture-of-Experts
• Semantic Specialization
• Efficient Adaptation
• Parameter-Efficient Fine-Tuning
• Domain Experts

Innovation

• Mixture-of-Experts
• Multimodal Adaptation
• Data-free Expert Identification
• Parameter-Efficient Fine-Tuning
• Semantic Specialization