conditional routing

This work investigates the information-theoretic efficiency of routing mechanisms in sparse Mixture-of-Experts (MoE) architectures, aiming to balance model accuracy with communication and computational resource utilization. The gating router is modeled as a stochastic channel, and a discrete mutual information estimator is proposed under a finite expert pool. Empirical posterior distributions \( q(W|S) \) are leveraged to compute \( I(X;T) \) and \( I(S;W) \), with the latter shown to exhibit a monotonic relationship with the generalization gap. The Blahut–Arimoto algorithm is employed to trace the accuracy–rate trade-off curve. Experiments demonstrate that the proposed mutual information estimator effectively tracks the generalization gap and significantly outperforms both the Xu–Raginsky bound and the uniform joint bound, offering a practical analytical tool for resource-aware MoE systems.

This work proposes a novel mixture-of-experts (MoE) architecture that eliminates the need for explicit routing mechanisms commonly found in traditional MoE models. By embedding activation logic directly within each expert and enabling end-to-end continuous gradient flow, experts autonomously determine their own activation without reliance on external routers, Softmax operations, Top-K selection, or hard-coded load-balancing heuristics. The approach introduces a unified, adaptive load-balancing framework that jointly optimizes resource allocation across both experts and tokens, supporting configurable dual-objective balancing. Experimental results demonstrate that the proposed model consistently outperforms existing baselines across multiple benchmarks, exhibiting superior scalability and robustness while removing rigid inductive biases imposed by centralized routing.

In conventional sparse Mixture-of-Experts (MoE) models, the Top-K routing mechanism is non-differentiable, impeding end-to-end optimization and limiting scalability. This paper proposes the first fully differentiable MoE architecture, replacing the discrete Top-K + Softmax routing with a continuous, ReLU-based differentiable routing mechanism. The method incorporates sparsity regularization and load-balancing constraints to ensure controlled expert activation and equitable expert utilization, while enabling dynamic expert selection and domain specialization. Extensive experiments across varying model scales, numbers of experts, and granularity levels demonstrate consistent and significant improvements over baseline Top-K MoE—particularly in expert scalability and downstream task performance. The architecture maintains high sparsity without sacrificing trainability or inference efficiency. Code is publicly available.

This work addresses two key limitations in mixture-of-experts (MoE) models: the lack of theoretical connection between MoE routing and self-attention, and the low sample efficiency of linear gating. We propose quadratic gating—replacing conventional linear routing with a quadratic function—and establish, for the first time, its rigorous equivalence to self-attention. Leveraging this equivalence, we derive principled design criteria for optimal quadratic gating and expert functions, leading to a novel high-performance attention mechanism. Theoretically, via statistical learning analysis, we prove that quadratic gating substantially enhances the expressivity and parameter/sample efficiency of expert selection. Empirically, our MoE variant outperforms linear-gating baselines across multiple tasks; the new attention mechanism surpasses state-of-the-art methods—including FlashAttention and Multi-Head Attention—while exhibiting strong alignment between theoretical predictions and empirical results. The framework thus achieves both interpretability and practical efficacy.

This work addresses the lack of a unified theoretical foundation in existing Mixture-of-Experts (MoE) models, where Top-k routing and load-balancing mechanisms are predominantly based on heuristics. From the perspectives of Bayesian inference and information theory, the authors establish a principled framework that rigorously proves MoE routing is equivalent to an NP-hard sparse subset selection problem. They introduce the notion of a “coherence barrier” and demonstrate that enhancing orthogonality among expert features substantially narrows the gap between greedy solutions and the global optimum. By integrating variational inference, information-theoretic entropy analysis, combinatorial optimization, and geometric orthogonality regularization, the study theoretically validates orthogonality regularization as the optimal engineering relaxation strategy for large-scale MoE systems, thereby providing a solid theoretical basis for the design and optimization of MoE architectures.

This work addresses the training challenges in Mixture-of-Experts (MoE) models caused by the non-differentiability of top-k routing. To overcome this, the authors propose ProbMoE, a framework that formulates expert selection as a probability distribution over discrete subsets under a cardinality constraint, thereby enabling differentiable routing. During forward propagation, exactly k experts are activated via constrained sampling, while backward propagation employs marginal probability gradients over the subset space as unbiased surrogates for true gradients. This approach enables, for the first time, probabilistic end-to-end training with exact k-expert routing and naturally extends to dynamic-k routing, allowing per-token adaptive expert assignment. Experiments demonstrate that the Exact-k variant significantly improves expert utilization and routing diversity, whereas the Dynamic-k variant achieves comparable performance with fewer activated experts.

This work addresses the “trust region collapse” problem in multi-model routing, where tight coupling between inference and routing leads to systematic suppression of high-capability experts. To resolve this, the authors propose EntroRouter, a single-round routing framework that decouples inference from routing by adopting entropy regularization as its core objective. The method introduces a high-entropy prior for exploration and a soft-anchor mechanism based on offline capability estimation, combined with soft-supervision initialization and a controlled entropy contraction strategy. These innovations effectively prevent strong experts from being underutilized. Experimental results demonstrate that EntroRouter reduces computational overhead by 48.25% while preserving 98.3% of the top expert’s accuracy.

This work addresses the inefficiency of traditional Mixture-of-Experts (MoE) architectures, which rely on static Top-k routing and cannot dynamically adjust the number of activated experts per input. Framing token routing as an information encoding task, the study establishes a novel connection between gating entropy and the Minimum Description Length (MDL) principle, leading to an uncertainty-aware adaptive routing mechanism that explicitly balances model complexity against performance. Evaluated across multiple backbone networks and benchmarks, the proposed method achieves a 36.5% improvement in expert activation sparsity while maintaining 99.5% of the original static routing performance, significantly outperforming existing static and heuristic dynamic routing strategies.

This work addresses the lack of effective design principles for routers in existing Mixture-of-Experts (MoE) models, which struggle to accurately capture the affinity between tokens and experts. The authors propose, for the first time, using the dominant singular directions of expert matrices as the target for router design and introduce a novel “power iteration followed by shrinkage” paradigm. During pretraining, they employ manifold optimization to dynamically align the router’s row vectors with these dominant singular directions. This approach achieves a favorable balance among alignment accuracy, computational efficiency, and training stability. Experiments on MoE models ranging from 1B to 11B parameters demonstrate substantial performance improvements, validating the effectiveness of the proposed router redesign strategy.

This study investigates how the routing mechanism of the Mixtral 8x7B-Instruct model influences safety outcomes in response to both benign and harmful prompts. By jointly analyzing expert activation frequencies and router gating gradients—and integrating targeted expert suppression with cross-group expert categorization—the work reveals, for the first time, the deep dependency and distributed nature of safety-related routing decisions. The findings demonstrate that safety-critical experts are broadly dispersed yet concentrated in specific layers; moreover, selectively suppressing experts identified via gradient-based importance significantly reduces restricted responses while inducing fewer side effects, thereby overcoming the limitations of single-metric analyses.