🤖 AI Summary
This study investigates the dynamic origins of expert load imbalance in Mixture-of-Experts (MoE) routing. By constructing a mean-field limit dynamical model for two-expert Softmax routing, the authors uncover the adaptive mechanisms and load evolution patterns inherent in the system. Theoretical analysis reveals that under symmetric conditions, the system undergoes a supercritical pitchfork bifurcation, while the introduction of external asymmetry induces a cusp catastrophe structure, offering a low-dimensional, controllable explanation for abrupt load imbalances. Leveraging bifurcation theory and the canonical form of the cusp catastrophe, the authors derive an exact parametric equation for the bifurcation set. Experimental validation using PyTorch with hard top-1 routing successfully reproduces the abrupt load shifts observed in real MoE systems, confirming the theoretical predictions.
📝 Abstract
We propose a minimal dynamical model of adaptive softmax routing for a two-expert Mixture-of-Experts (MoE) layer. The model is obtained as a mean-field limit of a discrete reinforcement rule: the selected expert receives a small score increment, while all scores undergo regularizing decay. In the symmetric case the limiting system has a supercritical pitchfork bifurcation: for weak feedback there is a unique stable balanced state, whereas above a critical feedback strength two stable asymmetric states appear. When an external asymmetry is added, the pitchfork unfolds into a pair of fold bifurcations forming a cusp in the control-parameter plane. We derive exact parametric equations for the bifurcation set and the local normal form of the cusp catastrophe. Numerical experiments connect this picture to empirical expert load, a small trainable MoE model, hard top-1 PyTorch routing, and a small classification experiment on digits. The results provide a controlled low-dimensional mechanism for abrupt transitions to load imbalance in adaptive MoE routers.