mixture of experts disentanglement

This paper addresses two critical challenges in large language model development: excessive computational overhead and difficulty in modeling heterogeneous, complex data. To tackle these, we present a systematic, up-to-date survey of Mixture-of-Experts (MoE) models. Unlike prior surveys—often outdated or narrowly scoped—we unify and analyze MoE advancements across emerging paradigms including continual learning, meta-learning, and reinforcement learning. We propose a comprehensive framework integrating theoretical analysis (e.g., convergence guarantees), multimodal adaptation (vision and language), and systems-level optimizations (sparse routing, load balancing, distributed training). Furthermore, we introduce a taxonomy of future research directions. Our work establishes the most complete MoE knowledge graph to date, explicitly identifying key bottlenecks and viable technical pathways. It serves as both a methodological foundation and an engineering roadmap for developing efficient, scalable large models.

本文通过局部聚合视角分析混合专家模型，探讨路由、稀疏激活和共享专家等设计选择的统计作用，分离出逼近误差、专家学习误差和路由器估计误差。

This work addresses the disconnect in existing Mixture-of-Experts (MoE) models between shared computation and dynamic routing, which overlooks the interdependence between reusable computation and residual expert requirements. The paper proposes UniF-MoE, a unified framework introducing a novel “shared-first, routed-later” mechanism: it first processes common features through a shared general-purpose module and then dynamically activates residual experts based on a shared-demand score and complementarity. Key innovations include key prototype selection, cumulative routing quality allocation, and Gram regularization to enhance routing sparsity and diversity, revealing a negative correlation between shared coverage and residual demand. Experiments demonstrate that UniF-MoE outperforms both static and dynamic MoE approaches on DomainBed and GLUE benchmarks while significantly reducing activated computation, inference latency, and memory footprint.

This work addresses the limitations of conventional sparse mixture-of-experts (MoE) models, which employ independent routing at each layer, resulting in an excessively large path space and poor statistical efficiency that hinder the learning of stable expert routing structures. To overcome this, the authors propose Path-Constrained Mixture of Experts (PathMoE), a novel architecture that shares router parameters across layers to dramatically reduce the effective path space, thereby enhancing path consistency and structural learnability. Notably, PathMoE naturally induces token clustering according to linguistic functionality without requiring auxiliary load-balancing losses. Experiments demonstrate that PathMoE achieves lower perplexity, superior downstream task performance, and greater robustness to routing perturbations compared to standard MoE baselines, consistently across both 0.9B and 16B parameter scales.

This study investigates the true origins of expert specialization in Mixture-of-Experts (MoE) models, challenging the assumption that routing mechanisms reflect genuine domain-specific expertise. Through theoretical analysis and empirical evaluation, the work demonstrates for the first time that expert usage similarity is entirely determined by the geometric structure of hidden states rather than architecture-induced specialization. It further reveals that load-balancing losses suppress directions corresponding to shared representations, leading to a phenomenon termed “specialization collapse.” Additionally, routing patterns in pretrained MoEs prove largely semantically uninterpretable. Combining linear mapping analysis, geometric modeling of hidden states, cross-model and cross-layer routing comparisons, and theoretical derivation of loss functions, the study validates across five pretrained models a strong alignment between routing behavior and representation-space geometry: expert activations exhibit high overlap even on semantically unrelated inputs, and prompt-level routing fails to predict expert selection during actual inference.

This work investigates the opaque expert specialization mechanism in Mixture-of-Experts (MoE) models, which limits inference and memory efficiency. By analyzing domain-specific routing patterns and employing an early-decoding framework, the study systematically examines how individual experts contribute to model outputs. Through comprehensive analyses—including routing distribution statistics, cosine similarity of hidden states, comparisons between single-expert and ensemble outputs, and perplexity evaluation—the authors find that a small subset of experts handles over 50% of all requests. Remarkably, outputs from a single dominant expert exhibit high consistency with the full model (cosine similarity up to 0.95), with only a 5% increase in perplexity. These findings suggest that precise expert pruning can substantially enhance inference efficiency without compromising performance, offering a promising avenue for efficient MoE deployment and knowledge localization.

This work systematically dissects the multidimensional design space of Mixture-of-Experts (MoE) architectures in large language models, moving beyond conventional generational narratives. It introduces a five-dimensional analytical framework encompassing expert granularity, topology, routing flexibility, load-balancing scope, and execution structure, and constructs a dependency graph to elucidate the coupling mechanisms across four control planes: expert topology, routing, load balancing, and expert parallelism. The framework’s validity is empirically demonstrated through iso-budget pretraining experiments integrating algorithmic innovations—such as Top-k routing, shared and fine-grained experts, and dynamic expert composition—with system-level optimizations including token dispatch, device placement, and all-to-all communication. The study further distills key open challenges for the future development of MoE systems.

为解决Mixture-of-Experts架构中专家专业化受限问题，提出对比路由机制（CoRM），通过对比每个token与隐藏状态的移动平均值来改进路由信号，提高零样本推理性能。

This work addresses a key limitation in sparse Mixture-of-Experts (MoE) models, where the routing mechanism jointly handles expert selection and output weighting, potentially constraining performance. The study provides the first systematic validation that these two functions should be decoupled and introduces Fixed Dispatch with Adaptive Aggregation (FDAA): a lightweight, learnable aggregation head is added atop a frozen backbone and fixed expert assignments, enabling end-to-end optimization of aggregation weights via the language modeling objective. Evaluated on pretrained MoE models such as OLMoE and DeepSeek-V2-Lite, FDAA achieves a 0.1523 reduction in cross-entropy on WikiText-103 and demonstrates consistent improvements across diverse benchmarks including C4 and PTB, confirming both the efficacy and generality of the proposed decoupling strategy.

This work addresses the training challenges in Mixture-of-Experts (MoE) models caused by the non-differentiability of top-k routing. To overcome this, the authors propose ProbMoE, a framework that formulates expert selection as a probability distribution over discrete subsets under a cardinality constraint, thereby enabling differentiable routing. During forward propagation, exactly k experts are activated via constrained sampling, while backward propagation employs marginal probability gradients over the subset space as unbiased surrogates for true gradients. This approach enables, for the first time, probabilistic end-to-end training with exact k-expert routing and naturally extends to dynamic-k routing, allowing per-token adaptive expert assignment. Experiments demonstrate that the Exact-k variant significantly improves expert utilization and routing diversity, whereas the Dynamic-k variant achieves comparable performance with fewer activated experts.

This work systematically investigates the interplay among key design dimensions in Mixture-of-Experts (MoE) architectures—such as the number of experts, expert granularity, heterogeneity, shared experts, and load balancing—through over 2,000 large-scale pretraining experiments. The study reveals that the number of experts and their granularity are the dominant factors governing model performance, while other design choices exert comparatively limited influence. Notably, increasing the total MoE parameters consistently enhances performance across all active parameter budgets, and the optimal expert size is determined solely by the number of active parameters. Furthermore, the effectiveness of dropless routing is empirically validated, demonstrating consistent performance gains.