mixture of experts

Existing PEFT methods neglect the dynamic expert routing mechanism inherent in Mixture-of-Experts (MoE) models, leading to architectural misalignment between adaptation modules and the underlying MoE structure. To address this, we propose *Routed-PEFT*, the first PEFT framework that explicitly incorporates expert routing into adapter design—dynamically assigning dedicated adapters to individual experts, thereby enabling joint optimization of expert specialization and task-specific adaptation. We systematically evaluate Routed-PEFT on OLMoE and Mixtral architectures, integrating it with LoRA, Adapter, and other PEFT variants under diverse routing strategies across commonsense and mathematical reasoning benchmarks. With only 0.1%–0.5% additional trainable parameters, Routed-PEFT achieves average accuracy gains of 2.3–5.7 percentage points over standard PEFT baselines. Moreover, our analysis uncovers task-dependent optimal routing configurations, establishing a novel paradigm for efficient fine-tuning of MoE models.

MoE models suffer from two key bottlenecks: imbalanced expert activation—causing parallel idle time and low utilization—and high communication overhead in expert parallelism. This paper proposes Collaborative-Constrained Routing (C2R), the first framework to establish an “expert collaboration–specialization” analytical paradigm. C2R explicitly regulates expert activation patterns via a collaboration-aware routing loss, a quantifiable expert specialization metric, lightweight gating regularization, and All2All communication optimization compatible with MegaBlocks—all without sacrificing model accuracy. Evaluated on LLaMA-MoE and Qwen-MoE, C2R improves downstream task performance by 0.51% and 0.33%, respectively, while significantly reducing inter-GPU All2All communication costs. Total training time decreases by 20–30% over state-of-the-art methods, achieving joint optimization of load balancing and communication efficiency.

This study investigates the true origins of expert specialization in Mixture-of-Experts (MoE) models, challenging the assumption that routing mechanisms reflect genuine domain-specific expertise. Through theoretical analysis and empirical evaluation, the work demonstrates for the first time that expert usage similarity is entirely determined by the geometric structure of hidden states rather than architecture-induced specialization. It further reveals that load-balancing losses suppress directions corresponding to shared representations, leading to a phenomenon termed “specialization collapse.” Additionally, routing patterns in pretrained MoEs prove largely semantically uninterpretable. Combining linear mapping analysis, geometric modeling of hidden states, cross-model and cross-layer routing comparisons, and theoretical derivation of loss functions, the study validates across five pretrained models a strong alignment between routing behavior and representation-space geometry: expert activations exhibit high overlap even on semantically unrelated inputs, and prompt-level routing fails to predict expert selection during actual inference.

This work addresses the severe expert routing imbalance in Expert Parallelism (EP) during inference or post-training, which often overloads certain devices and creates computational and memory bottlenecks. To tackle this issue, the paper introduces a dynamic rerouting mechanism—the first of its kind for non-uniform MoE routing—that migrates tokens and expert parameters from overloaded to underutilized devices while preserving model expressiveness, thereby minimizing latency and satisfying memory constraints. This approach explicitly relaxes the implicit assumption of routing balance in conventional EP and integrates a hardware-aware hyperparameter tuning framework to significantly enhance deployment efficiency. Experimental results demonstrate up to 5× speedup and a 4× reduction in peak memory usage compared to standard EP, with a 1.9× improvement in inference throughput on the gpt-oss-120b model.

This work addresses the slow convergence and training instability commonly observed in traditional Mixture-of-Experts (MoE) models, which stem from the joint optimization of routing policies and expert weights. To overcome these limitations, the authors propose Grouter, a novel approach that introduces a preset routing mechanism: high-quality routing structures are distilled from a pre-trained MoE model and then fixed, effectively decoupling routing optimization from expert weight updates. Grouter further incorporates expert folding, expert fine-tuning, and structure-prior-guided training strategies to enable efficient adaptation across diverse model configurations and data distributions. Experimental results demonstrate that Grouter improves training data utilization by 4.28× and achieves up to 33.5% higher throughput, significantly enhancing both the efficiency and performance of MoE training.

Existing approaches struggle to disentangle load balancing from functional specialization in Mixture-of-Experts (MoE) models, hindering a deeper understanding of expert mechanisms. This work proposes DBES, a diagnostic framework that, for the first time, moves beyond accuracy-based evaluation and systematically quantifies the degree of expert functional specialization through five theoretically grounded metrics: Routing Specialization, Normalized Effective Rank, Domain Isolation, Routing Stiffness Score, and N-gram Expertise. Leveraging this diagnostic insight, we design an intervention-based post-training strategy that, using only 15% of the original training resources, selectively activates highly specialized expert pathways, achieving performance gains of 66% to 94.48% on domain-specific tasks. Our analysis further reveals distinct specialization paradigms across models and offers actionable pathways for targeted optimization.

This study investigates the relationship between safety behaviors and expert routing mechanisms in aligned Mixture-of-Experts (MoE) large language models. The authors find that a model’s safety capabilities can be concentrated in a small subset of experts and are largely independent of the routing policy, rather than being driven by dedicated refusal-oriented experts. To leverage this insight, they propose the Router-Agnostic Safety-critical Expert Tuning (RASET) framework, which integrates a contrastive routing sensitivity criterion with parameter-efficient fine-tuning to precisely identify and optimize safety-critical experts without altering the original routing behavior. Experiments demonstrate that RASET significantly steers model safety outputs with minimal semantic interference, revealing for the first time the existence of localized, manipulable expert-level safety mechanisms—and their potential vulnerabilities—within MoE architectures.

This work addresses the lack of effective design principles for routers in existing Mixture-of-Experts (MoE) models, which struggle to accurately capture the affinity between tokens and experts. The authors propose, for the first time, using the dominant singular directions of expert matrices as the target for router design and introduce a novel “power iteration followed by shrinkage” paradigm. During pretraining, they employ manifold optimization to dynamically align the router’s row vectors with these dominant singular directions. This approach achieves a favorable balance among alignment accuracy, computational efficiency, and training stability. Experiments on MoE models ranging from 1B to 11B parameters demonstrate substantial performance improvements, validating the effectiveness of the proposed router redesign strategy.

This work addresses the pervasive issue of deep routing collapse in large Mixture-of-Experts (MoE) models for low-resource languages, which leads to imbalanced expert utilization and constrained multilingual capabilities. The study reveals, for the first time, that this phenomenon stems from insufficient pretraining data rather than inherent linguistic properties. To diagnose multilingual capacity, the authors propose routing entropy and expert specialization as key indicators. Through balanced bilingual continual pretraining (CPT) and supervised fine-tuning (SFT) on Hebrew, Japanese, and other languages using both pure Transformer and Mamba-Transformer hybrid architectures, they demonstrate that CPT substantially increases routing entropy, encourages language-agnostic expert sharing, and consistently enhances downstream performance, whereas SFT yields limited gains. These findings underscore the critical role of data balance in scaling MoE models multilingually.

This work addresses the sensitivity of Mixture-of-Experts (MoE) models to parameter perturbations during model merging, which often leads to routing collapse and severe performance degradation. The study is the first to identify this issue and introduces Hessian-aware Router Calibration (HARC), a training-free framework that leverages second-order curvature information from the Hessian matrix to analytically realign the router post-merging, thereby restoring its routing capability. By integrating matrix-free conjugate gradient methods with Top-k routing analysis, HARC significantly enhances the performance of various MoE merging baselines on mathematical reasoning and code generation tasks, effectively mitigating routing failure without additional training.