token routing

This work systematically investigates the interplay among key design dimensions in Mixture-of-Experts (MoE) architectures—such as the number of experts, expert granularity, heterogeneity, shared experts, and load balancing—through over 2,000 large-scale pretraining experiments. The study reveals that the number of experts and their granularity are the dominant factors governing model performance, while other design choices exert comparatively limited influence. Notably, increasing the total MoE parameters consistently enhances performance across all active parameter budgets, and the optimal expert size is determined solely by the number of active parameters. Furthermore, the effectiveness of dropless routing is empirically validated, demonstrating consistent performance gains.

This work challenges the prevailing assumption that Mixture-of-Experts (MoE) models achieve domain specialization through sparse routing, introducing the COMMITTEEAUDIT framework to systematically analyze expert-level routing behavior. Through quantitative and qualitative evaluation of multiple representative MoE models on the MMLU benchmark, we uncover the existence of persistent “standing committees”—a small subset of experts that consistently dominate routing weights across domains and layers, regardless of routing budget constraints. These core experts anchor structural and syntactic reasoning, while peripheral experts handle only narrow, domain-specific knowledge. Our findings reveal that the actual degree of specialization in MoE models is substantially lower than commonly assumed and suggest that current load-balancing training objectives may conflict with the model’s intrinsic optimization dynamics.

This work addresses the fundamental trade-off in sparse Mixture-of-Experts (MoE) models between load balancing and expert specialization, which often leads to routing collapse or diminished expert diversity. The authors propose Hi-MoE, a novel framework that decomposes routing into two coupled hierarchical levels: inter-group routing ensures balanced token distribution across expert groups, while intra-group routing fosters complementary expert specialization and prevents collapse. This principled redesign of router behavior consistently outperforms existing sparse routing and grouped MoE approaches across both NLP and vision benchmarks. In a 58B-token pretraining setting, Hi-MoE-7B achieves a 5.6% lower perplexity and 40% improved expert balance compared to OLMoE-7B.

This work proposes a novel mixture-of-experts (MoE) architecture that eliminates the need for explicit routing mechanisms commonly found in traditional MoE models. By embedding activation logic directly within each expert and enabling end-to-end continuous gradient flow, experts autonomously determine their own activation without reliance on external routers, Softmax operations, Top-K selection, or hard-coded load-balancing heuristics. The approach introduces a unified, adaptive load-balancing framework that jointly optimizes resource allocation across both experts and tokens, supporting configurable dual-objective balancing. Experimental results demonstrate that the proposed model consistently outperforms existing baselines across multiple benchmarks, exhibiting superior scalability and robustness while removing rigid inductive biases imposed by centralized routing.

This work investigates the opaque expert specialization mechanism in Mixture-of-Experts (MoE) models, which limits inference and memory efficiency. By analyzing domain-specific routing patterns and employing an early-decoding framework, the study systematically examines how individual experts contribute to model outputs. Through comprehensive analyses—including routing distribution statistics, cosine similarity of hidden states, comparisons between single-expert and ensemble outputs, and perplexity evaluation—the authors find that a small subset of experts handles over 50% of all requests. Remarkably, outputs from a single dominant expert exhibit high consistency with the full model (cosine similarity up to 0.95), with only a 5% increase in perplexity. These findings suggest that precise expert pruning can substantially enhance inference efficiency without compromising performance, offering a promising avenue for efficient MoE deployment and knowledge localization.

This work addresses the challenges of weak generalization, expert redundancy, and degraded performance on rare domains in multi-dataset joint training, which stem from distribution shifts and semantic inconsistencies in labels. To this end, the authors propose the GEM framework, which replaces the learnable router in conventional Mixture-of-Experts (MoE) architectures with a planner–compiler design. GEM introduces a deterministic expert assignment mechanism based on linear programming relaxation and hierarchical rounding, strictly enforcing capacity constraints without relying on load-balancing losses. This approach significantly enhances expert specialization and routing interpretability. Integrated with a DINO vision backbone, GEM-DINO achieves state-of-the-art performance on the UODB benchmark, markedly improving results on rare datasets and effectively mitigating task interference in few-shot adaptation scenarios.

This work addresses the limitations of conventional sparse mixture-of-experts (MoE) models, which employ independent routing at each layer, resulting in an excessively large path space and poor statistical efficiency that hinder the learning of stable expert routing structures. To overcome this, the authors propose Path-Constrained Mixture of Experts (PathMoE), a novel architecture that shares router parameters across layers to dramatically reduce the effective path space, thereby enhancing path consistency and structural learnability. Notably, PathMoE naturally induces token clustering according to linguistic functionality without requiring auxiliary load-balancing losses. Experiments demonstrate that PathMoE achieves lower perplexity, superior downstream task performance, and greater robustness to routing perturbations compared to standard MoE baselines, consistently across both 0.9B and 16B parameter scales.

This work addresses the training challenges in Mixture-of-Experts (MoE) models caused by the non-differentiability of top-k routing. To overcome this, the authors propose ProbMoE, a framework that formulates expert selection as a probability distribution over discrete subsets under a cardinality constraint, thereby enabling differentiable routing. During forward propagation, exactly k experts are activated via constrained sampling, while backward propagation employs marginal probability gradients over the subset space as unbiased surrogates for true gradients. This approach enables, for the first time, probabilistic end-to-end training with exact k-expert routing and naturally extends to dynamic-k routing, allowing per-token adaptive expert assignment. Experiments demonstrate that the Exact-k variant significantly improves expert utilization and routing diversity, whereas the Dynamic-k variant achieves comparable performance with fewer activated experts.

Existing approaches struggle to disentangle load balancing from functional specialization in Mixture-of-Experts (MoE) models, hindering a deeper understanding of expert mechanisms. This work proposes DBES, a diagnostic framework that, for the first time, moves beyond accuracy-based evaluation and systematically quantifies the degree of expert functional specialization through five theoretically grounded metrics: Routing Specialization, Normalized Effective Rank, Domain Isolation, Routing Stiffness Score, and N-gram Expertise. Leveraging this diagnostic insight, we design an intervention-based post-training strategy that, using only 15% of the original training resources, selectively activates highly specialized expert pathways, achieving performance gains of 66% to 94.48% on domain-specific tasks. Our analysis further reveals distinct specialization paradigms across models and offers actionable pathways for targeted optimization.

This work addresses the substantial cross-node communication overhead in multi-node Mixture-of-Experts (MoE) inference, which stems from imbalanced expert loads and inefficient token routing. For the first time, it systematically characterizes three key properties of MoE expert activation: dynamic load imbalance, task-domain-dependent expert preferences, and strong correlation between the prefill and decode phases. Leveraging these insights, the authors propose a workload-aware microbatch grouping and expert placement strategy that enhances token-expert locality. Evaluated on over 100,000 real-world activation traces across multiple MoE models and datasets, the approach reduces all-to-all communication volume by up to 20×, significantly lowering decoding latency and improving accelerator utilization.