🤖 AI Summary
It remains unclear whether the routing mechanism in sparse Mixture-of-Experts (MoE) models exhibits task-conditioned behavior. This work proposes the concept of “routing signatures” to characterize the activation patterns of experts across layers for different tasks and provides the first empirical evidence that MoE routing is highly task-sensitive. By constructing routing signatures, defining similarity metrics, applying logistic regression classification, and analyzing inter-layer routing signals, the study reveals that routing signatures of tasks within the same category exhibit a similarity of 0.8435, significantly higher than the 0.6225 observed across categories (Cohen’s d = 1.44). Remarkably, routing signatures alone achieve 92.5% accuracy in four-way task classification. The authors release the MOE-XRAY toolkit to advance interpretability research in MoE models, establishing a new paradigm for analyzing expert routing dynamics.
📝 Abstract
Sparse Mixture-of-Experts (MoE) architectures enable efficient scaling of large language models through conditional computation, yet the routing mechanisms responsible for expert selection remain poorly understood. In this work, we introduce routing signatures, a vector representation summarizing expert activation patterns across layers for a given prompt, and use them to study whether MoE routing exhibits task-conditioned structure. Using OLMoE-1B-7B-0125-Instruct as an empirical testbed, we show that prompts from the same task category induce highly similar routing signatures, while prompts from different categories exhibit substantially lower similarity. Within-category routing similarity (0.8435 +/- 0.0879) significantly exceeds across-category similarity (0.6225 +/- 0.1687), corresponding to Cohen's d = 1.44. A logistic regression classifier trained solely on routing signatures achieves 92.5% +/- 6.1% cross-validated accuracy on four-way task classification. To ensure statistical validity, we introduce permutation and load-balancing baselines and show that the observed separation is not explained by sparsity or balancing constraints alone. We further analyze layer-wise signal strength and low-dimensional projections of routing signatures, finding that task structure becomes increasingly apparent in deeper layers. These results suggest that routing in sparse transformers is not merely a balancing mechanism, but a measurable task-sensitive component of conditional computation. We release MOE-XRAY, a lightweight toolkit for routing telemetry and analysis.