🤖 AI Summary
This work addresses the lack of a unified theoretical foundation in existing Mixture-of-Experts (MoE) models, where Top-k routing and load-balancing mechanisms are predominantly based on heuristics. From the perspectives of Bayesian inference and information theory, the authors establish a principled framework that rigorously proves MoE routing is equivalent to an NP-hard sparse subset selection problem. They introduce the notion of a “coherence barrier” and demonstrate that enhancing orthogonality among expert features substantially narrows the gap between greedy solutions and the global optimum. By integrating variational inference, information-theoretic entropy analysis, combinatorial optimization, and geometric orthogonality regularization, the study theoretically validates orthogonality regularization as the optimal engineering relaxation strategy for large-scale MoE systems, thereby providing a solid theoretical basis for the design and optimization of MoE architectures.
📝 Abstract
Mixture-of-Experts models enable large language models to scale efficiently, as they only activate a subset of experts for each input. Their core mechanisms, Top-k routing and auxiliary load balancing, remain heuristic, however, lacking a cohesive theoretical underpinning to support them. To this end, we build the first unified theoretical framework that rigorously derives these practices as optimal sparse posterior approximation and prior regularization from a Bayesian perspective, while simultaneously framing them as mechanisms to minimize routing ambiguity and maximize channel capacity from an information-theoretic perspective. We also pinpoint the inherent combinatorial hardness of routing, defining it as the NP-hard sparse subset selection problem. We rigorously prove the existence of a"Coherence Barrier"; when expert representations exhibit high mutual coherence, greedy routing strategies theoretically fail to recover the optimal expert subset. Importantly, we formally verify that imposing geometric orthogonality in the expert feature space is sufficient to narrow the divide between the NP-hard global optimum and polynomial-time greedy approximation. Our comparative analyses confirm orthogonality regularization as the optimal engineering relaxation for large-scale models. Our work offers essential theoretical support and technical assurance for a deeper understanding and novel designs of MoE.