multi-granularity moe

This work addresses a critical gap in existing literature by providing the first comprehensive survey of sparse Mixture-of-Experts (MoE) models, systematically integrating their algorithmic foundations, decentralized architectures, and applications in vertical domains. It thoroughly examines core mechanisms such as routing strategies and expert network design, while further extending the discussion to decentralized deployment paradigms and adaptation methods for cross-modal and domain-specific scenarios. By synthesizing recent advances across these dimensions, this survey fills a notable void in the current body of review literature and offers an authoritative reference for researchers and practitioners aiming to develop efficient, scalable large models grounded in sparse MoE principles.

本文通过局部聚合视角分析混合专家模型，探讨路由、稀疏激活和共享专家等设计选择的统计作用，分离出逼近误差、专家学习误差和路由器估计误差。

The internal mechanisms and modular nature of Mixture-of-Experts (MoE) large language models remain poorly understood, particularly regarding expert granularity, routing behavior, and layer-wise expert diversity. Method: We conduct attribution analysis, expert activation visualization, output norm statistics, and controlled experiments across three representative MoE architectures—Mixtral, GLaM, and DeepSpeed-MoE. Contribution/Results: We empirically establish that individual neurons function as fine-grained experts; routers exhibit strong preference for high-norm experts; and expert diversity generally increases with network depth—except in the final layer. Based on these findings, we formulate a hierarchical evolution law of expert diversity and provide actionable guidelines for router design and expert allocation. Our work formally validates the modular architecture of MoE models, identifies anomalous behavior in the top layer, and has directly informed routing strategy improvements across multiple research teams. The open-sourced code has garnered significant community attention.

This work investigates the opaque expert specialization mechanism in Mixture-of-Experts (MoE) models, which limits inference and memory efficiency. By analyzing domain-specific routing patterns and employing an early-decoding framework, the study systematically examines how individual experts contribute to model outputs. Through comprehensive analyses—including routing distribution statistics, cosine similarity of hidden states, comparisons between single-expert and ensemble outputs, and perplexity evaluation—the authors find that a small subset of experts handles over 50% of all requests. Remarkably, outputs from a single dominant expert exhibit high consistency with the full model (cosine similarity up to 0.95), with only a 5% increase in perplexity. These findings suggest that precise expert pruning can substantially enhance inference efficiency without compromising performance, offering a promising avenue for efficient MoE deployment and knowledge localization.

Existing PEFT methods neglect the dynamic expert routing mechanism inherent in Mixture-of-Experts (MoE) models, leading to architectural misalignment between adaptation modules and the underlying MoE structure. To address this, we propose *Routed-PEFT*, the first PEFT framework that explicitly incorporates expert routing into adapter design—dynamically assigning dedicated adapters to individual experts, thereby enabling joint optimization of expert specialization and task-specific adaptation. We systematically evaluate Routed-PEFT on OLMoE and Mixtral architectures, integrating it with LoRA, Adapter, and other PEFT variants under diverse routing strategies across commonsense and mathematical reasoning benchmarks. With only 0.1%–0.5% additional trainable parameters, Routed-PEFT achieves average accuracy gains of 2.3–5.7 percentage points over standard PEFT baselines. Moreover, our analysis uncovers task-dependent optimal routing configurations, establishing a novel paradigm for efficient fine-tuning of MoE models.

This work addresses the disconnect in existing Mixture-of-Experts (MoE) models between shared computation and dynamic routing, which overlooks the interdependence between reusable computation and residual expert requirements. The paper proposes UniF-MoE, a unified framework introducing a novel “shared-first, routed-later” mechanism: it first processes common features through a shared general-purpose module and then dynamically activates residual experts based on a shared-demand score and complementarity. Key innovations include key prototype selection, cumulative routing quality allocation, and Gram regularization to enhance routing sparsity and diversity, revealing a negative correlation between shared coverage and residual demand. Experiments demonstrate that UniF-MoE outperforms both static and dynamic MoE approaches on DomainBed and GLUE benchmarks while significantly reducing activated computation, inference latency, and memory footprint.

This work systematically investigates the interplay among key design dimensions in Mixture-of-Experts (MoE) architectures—such as the number of experts, expert granularity, heterogeneity, shared experts, and load balancing—through over 2,000 large-scale pretraining experiments. The study reveals that the number of experts and their granularity are the dominant factors governing model performance, while other design choices exert comparatively limited influence. Notably, increasing the total MoE parameters consistently enhances performance across all active parameter budgets, and the optimal expert size is determined solely by the number of active parameters. Furthermore, the effectiveness of dropless routing is empirically validated, demonstrating consistent performance gains.

本文提出MetaNet，通过预测每层的专家保留阈值和路由偏置来动态分配专家，解决了MoE模型中固定专家数量导致的效率问题。

This work addresses the fundamental trade-off in sparse Mixture-of-Experts (MoE) models between load balancing and expert specialization, which often leads to routing collapse or diminished expert diversity. The authors propose Hi-MoE, a novel framework that decomposes routing into two coupled hierarchical levels: inter-group routing ensures balanced token distribution across expert groups, while intra-group routing fosters complementary expert specialization and prevents collapse. This principled redesign of router behavior consistently outperforms existing sparse routing and grouped MoE approaches across both NLP and vision benchmarks. In a 58B-token pretraining setting, Hi-MoE-7B achieves a 5.6% lower perplexity and 40% improved expert balance compared to OLMoE-7B.

This work addresses the pervasive issue of deep routing collapse in large Mixture-of-Experts (MoE) models for low-resource languages, which leads to imbalanced expert utilization and constrained multilingual capabilities. The study reveals, for the first time, that this phenomenon stems from insufficient pretraining data rather than inherent linguistic properties. To diagnose multilingual capacity, the authors propose routing entropy and expert specialization as key indicators. Through balanced bilingual continual pretraining (CPT) and supervised fine-tuning (SFT) on Hebrew, Japanese, and other languages using both pure Transformer and Mamba-Transformer hybrid architectures, they demonstrate that CPT substantially increases routing entropy, encourages language-agnostic expert sharing, and consistently enhances downstream performance, whereas SFT yields limited gains. These findings underscore the critical role of data balance in scaling MoE models multilingually.

Existing research on vision-based Mixture-of-Experts (MoE) models predominantly relies on category-level routing statistics, which obscures the actual representational content encoded by individual experts. This work trains sparsely gated convolutional MoE models and advances expert analysis from categorical labels to continuous visual and semantic feature dimensions for the first time. By integrating contrastive learning, neuroscience-inspired tuning analyses, and representational similarity analysis (RSA)—augmented with human semantic judgments from the THINGS dataset to define semantic axes—we demonstrate that experts consistently differentiate along continuous semantic dimensions such as “animate–inanimate.” Despite sparse routing, experts collectively span a broad semantic space. While experts exhibit comparable category discriminability, their feature tuning profiles differ markedly, underscoring the necessity and efficacy of expert-level representational analysis.