MoE$^2$-LoRA: When MoE Models Meet MoE-style Low-Rank Adaptation

2026-07-24

AI Summary

Existing parameter-efficient fine-tuning (PEFT) methods for Mixture-of-Experts (MoE) models struggle to simultaneously leverage routing priors, enable dynamic adaptation, and facilitate cross-expert knowledge sharing, often resulting in suboptimal efficiency, high catastrophic forgetting risk, or constrained model capacity. To address these limitations, this work proposes MoE²-LoRA, the first approach that integrates the MoE paradigm into LoRA-based fine-tuning. It introduces a Routing-Conditioned Projection (RCP) module that aligns pre-trained expert specialization with task-specific adaptation and constructs a globally shared pool of LoRA experts. This design enables dynamic low-rank adaptation guided by the original router activations and promotes cross-layer knowledge sharing. Evaluated across MoE backbones of varying scales and granularities, MoE²-LoRA achieves state-of-the-art performance on downstream tasks while demonstrating superior generalization capabilities.

Abstract

Mixture-of-Experts (MoE) architectures have been widely adopted in large language models, yet parameter-efficient fine-tuning (PEFT) for MoE models remains underexplored. Existing PEFT methods for MoE either ignore router priors with uniform adapters, reducing efficiency and risking forgetting, or rely on static expert selection, limiting per-token capacity and cross-expert feature learning. In this paper, we make the first attempt to fine-tune MoE models with MoE-style low-rank adaptation: our method, entitled MoE$^2$-LoRA, deeply couples the pretrained expert specialization with task-specific adaptivity via a dual-channel Routing-Conditioned Projection (RCP) module, which reuses base router activations to inform LoRA routing. We further introduce a single global LoRA expert pool shared across all layers, enabling model-wide adaptation with emergent layer-wise affinities and balanced expert utilization. MoE$^2$-LoRA simultaneously benefits from the advantages of prior reuse, dynamic adapter routing, and model-wide knowledge sharing. Evaluated on multiple MoE backbones with varying scales and expert granularities, MoE$^2$-LoRA consistently achieves state-of-the-art downstream accuracy while retaining stronger general capabilities.

Problem

• Mixture-of-Experts
• parameter-efficient fine-tuning
• expert routing
• low-rank adaptation
• MoE models

Innovation

• Mixture-of-Experts
• LoRA
• parameter-efficient fine-tuning
• dynamic routing
• shared expert pool