DistMoE: Private-data Rehearsal-free Routing in Mixture-of-Experts for Distributed Instruction Tuning

2026-08-10

AI Summary

This work addresses the challenge of efficiently performing distributed instruction tuning for multimodal large language models when training data are scattered across multiple private, permission-restricted clients. To this end, the authors propose DistMoE, a method that integrates client-private experts with shared feedforward networks within each layer of the language decoder. DistMoE introduces a token-level routing mechanism that operates without replaying private client data, and combines isotropic regularization loss with lightweight projection adapters to enable modular, label-agnostic cross-client domain adaptation. Experimental results demonstrate that DistMoE achieves effective domain adaptation and expert reuse across multiple vision–language benchmarks, delivering competitive performance while preserving the privacy of client-specific knowledge.

Abstract

Multimodal Large Language Models (MLLMs) have shown strong multimodal instruction-following ability, but adapting them to diverse visual-language domains typically assumes centralized data access and costly joint training. This is restrictive when data is distributed across private, domain-specific, or permission-limited clients. To this end, we propose DistMoE, a mixture-of-experts (MoE) approach for distributed visual instruction tuning. In each layer of the language decoder it augments the public feedforward network (FFN) with a client-specific private FFN expert, with the goal to acquire domain-specific knowledge. However, independent expert training causes the private FFNs to learn representation of different scale and magnitudes, making merging the experts difficult. To reduce client-specific drift, we introduce a public-anchored expert composition stage that updates only routers and lightweight private projection adapters on a mix of local client data and public data, via an isotropic regularization loss, therefore making it cross-client rehearsal-free composition. During inference, DistMoE performs modular routing over public and private experts, enabling token-wise domain composition without explicit domain labels. Experiments across diverse visual-language benchmarks show that DistMoE enables flexible expert reuse, effective domain adaptation, and competitive performance while preserving modular control over client-specific knowledge. Codes are available at https://github.com/mainaksingha01/DistMoE.

Problem

• distributed instruction tuning
• mixture-of-experts
• private data
• rehearsal-free
• multimodal LLMs

Innovation

• Mixture-of-Experts
• Distributed Instruction Tuning
• Private-data Rehearsal-free Routing
• Modular Routing
• Isotropic Regularization