train gating networks

This work investigates the trade-off between model expressivity and generalization performance in Mixture-of-Experts (MoE) architectures under communication constraints. For the first time, rate-distortion theory is introduced into MoE analysis by modeling the gating mechanism as a stochastic channel operating at a finite rate. By integrating mutual information-based generalization bounds with the rate-distortion function \(D(R_g)\), the study establishes a quantitative relationship between the gating communication rate and generalization error. A theoretical upper bound on generalization error is derived and validated through synthetic multi-expert model simulations, which demonstrate that reducing the gating rate, while limiting expressivity, can enhance generalization. Based on these insights, the paper proposes a capacity-aware design principle for MoE systems, offering theoretical guidance for efficient model construction in resource-constrained settings.

This work addresses two key limitations in mixture-of-experts (MoE) models: the lack of theoretical connection between MoE routing and self-attention, and the low sample efficiency of linear gating. We propose quadratic gating—replacing conventional linear routing with a quadratic function—and establish, for the first time, its rigorous equivalence to self-attention. Leveraging this equivalence, we derive principled design criteria for optimal quadratic gating and expert functions, leading to a novel high-performance attention mechanism. Theoretically, via statistical learning analysis, we prove that quadratic gating substantially enhances the expressivity and parameter/sample efficiency of expert selection. Empirically, our MoE variant outperforms linear-gating baselines across multiple tasks; the new attention mechanism surpasses state-of-the-art methods—including FlashAttention and Multi-Head Attention—while exhibiting strong alignment between theoretical predictions and empirical results. The framework thus achieves both interpretability and practical efficacy.

This work addresses the lack of a unified computational interpretation for neural policy gating mechanisms. We propose GateMod, a theoretically grounded gating framework that couples task structure with neural circuit dynamics via the principle of free-energy minimization. GateMod comprises two core components: GateFlow—a continuous-time energy-flow model—and GateNet—a soft-competitive recurrent network—enabling emergent gating for skill composition and behavioral planning. We formally prove GateMod’s global exponential convergence and robustness under perturbations. Empirically, GateMod achieves significant performance gains over state-of-the-art methods in multi-agent cooperative tasks and human multi-armed bandit experiments. Crucially, it provides the first quantitative demonstration of how task structure modulates gating behavior through neural energy dynamics. By offering a computationally precise and empirically testable account, GateMod establishes a principled theoretical foundation for understanding strategy selection in prefrontal–basal ganglia circuits.

This work addresses unresolved challenges in sigmoid-gated Mixture-of-Experts (MoE) models for classification tasks—namely, poor convergence, low sample efficiency, and undesirable coupling between the temperature parameter and gating dynamics. The authors propose an improved sigmoid gating mechanism that, for the first time, provably outperforms softmax gating in multi-class settings. By replacing the inner product with a Euclidean distance-based scoring function, the method effectively decouples the temperature from gating parameters, leading to markedly improved optimization dynamics. Theoretical analysis demonstrates that the approach reduces sample complexity from exponential to polynomial in both expert selection and parameter estimation, substantially lowering the data requirements and thereby enhancing model scalability and training efficiency.

This work investigates how Softmax-based gating mechanisms affect parameter estimation and convergence rates in Mixture-of-Experts (MoE) models. We establish, for the first time, unified theoretical convergence bounds for three gating architectures: standard Softmax, sparsified Softmax, and hierarchical Softmax. Introducing the notion of *strong identifiability*, we prove that two-layer nonlinear experts are identifiable from polynomially many samples, whereas linear experts suffer from parameter coupling constrained by partial differential equations, necessitating exponentially many samples—thereby revealing a fundamental trade-off between expert structure identifiability and sample complexity. By integrating convergence analysis, identifiability theory, and statistical learning principles, we quantitatively characterize the interplay between gating design and sample efficiency. Our results fill a critical theoretical gap in MoE gating mechanisms and provide rigorous foundations for designing computationally efficient, statistically sound MoE architectures.

This work addresses the lack of systematic theoretical understanding of the posterior behavior of Bayesian Softmax-gated Mixture-of-Experts models in density estimation, parameter estimation, and expert number selection. It establishes, for the first time, posterior contraction rate theory for this model, providing rigorous guarantees for density estimation under both fixed and varying numbers of experts, and proving consistency of parameter estimation. To handle the model’s intricate identifiability structure, the study introduces a Voronoi-type loss function and develops two complementary Bayesian strategies for selecting the number of experts. These contributions offer foundational theoretical insights and practical guidance for nonparametric Bayesian Mixture-of-Experts models.

This work investigates the information-theoretic efficiency of routing mechanisms in sparse Mixture-of-Experts (MoE) architectures, aiming to balance model accuracy with communication and computational resource utilization. The gating router is modeled as a stochastic channel, and a discrete mutual information estimator is proposed under a finite expert pool. Empirical posterior distributions \( q(W|S) \) are leveraged to compute \( I(X;T) \) and \( I(S;W) \), with the latter shown to exhibit a monotonic relationship with the generalization gap. The Blahut–Arimoto algorithm is employed to trace the accuracy–rate trade-off curve. Experiments demonstrate that the proposed mutual information estimator effectively tracks the generalization gap and significantly outperforms both the Xu–Raginsky bound and the uniform joint bound, offering a practical analytical tool for resource-aware MoE systems.

This work proposes the first cost-aware routing framework for supervised fine-tuning data acquisition that integrates statistical gating with an adversarial adjudication mechanism to efficiently identify high-value corpora while avoiding costly misacquisitions. The approach evaluates candidate samples along three axes—diversity, utility, and redundancy—using low-cost statistical estimates for initial filtering and triggering a multi-agent debate between proponent and opponent advocates only when confidence is insufficient. Evaluated through quality assessments with confidence intervals and controlled synthetic benchmarks, the system achieves 0.90 accuracy and 0.83 F₁ score across twelve datasets at a unit cost of just $0.017, substantially outperforming always-verify strategies. Moreover, it provides the first quantitative evidence of stance bias (52% stance reversal) and oppositional advantage (80% win rate) in LLM-based adjudication.

This work addresses the challenge of enabling input-dependent conditional computation during inference while simultaneously achieving effective training regularization and computational efficiency. The authors propose DynamicGate-MLP, a framework that unifies Dropout-style regularization with conditional computation through a learnable continuous gating mechanism. During training, expected gating values provide regularization, while at inference time, the Straight-Through Estimator yields discrete execution paths that dynamically activate subnetworks. A compute budget constraint based on expected gate utilization is introduced, and layer-weighted relative MACs are used to evaluate efficiency. Experiments across multiple datasets—including MNIST, CIFAR-10, Tiny-ImageNet, Speech Commands, and PBMC3k—demonstrate that the method significantly reduces computational overhead while maintaining competitive performance.