🤖 AI Summary
This work addresses the lack of interpretability and controllability in expert specialization mechanisms within sparse Mixture-of-Experts (MoE) models. The authors propose a low-dimensional geometric routing method based on cosine similarity, which endows each rank-1 expert with inherent mono-semanticity. For the first time, expert-level specialization is established as a fundamental unit of interpretability, exhibiting structural mono-semanticity, causal verifiability, and zero-overhead inference controllability. Experiments demonstrate that 15% of experts can be explicitly mapped to 10 distinct semantic categories. Furthermore, causal interventions enable significant modulation of target class output probabilities—e.g., increasing temporal category likelihood by 321% or decreasing geographic category likelihood by 23%—with effects that accumulate across multiple layers.
📝 Abstract
Sparse Mixture-of-Experts (MoE) models scale parameters while fixing active computation per token, but the specialization of individual experts remains opaque. In a companion paper we showed that routing topology is quality-neutral: five structurally different configurations converge to statistically equivalent language modeling quality. Here we show that expert identity is nonetheless causally meaningful: individual rank-1 experts are monosemantic by construction, and cosine-similarity routing in a low-dimensional metric space makes their specialization directly inspectable.
We present four lines of evidence. First, projecting expert output vectors through the unembedding matrix yields a Semantic Dictionary: 15% of experts are monosemantic specialists spanning 10 categories (temporal, geographic, cardinal, discourse, emotional, financial, military, scientific). Second, routing exhibits a frequency-to-syntax gradient: early layers separate tokens by word frequency, deeper layers by syntactic class (Zipf-confound controls, all $p < 0.001$). Third, causal interventions confirm these labels: steering toward a temporal expert's centroid increases P(temporal) by +321% (median across 44 prompts); suppressing a geographic expert drops P(geographic) by -23%; rewriting an expert's output vector halves target-category probability, and effects compose additively across layers. Fourth, the interventions are not unique to cosine routing: linear routers support comparable steering, but only cosine routing provides geometric transparency -- expert specialization is readable directly from the centroid matrix.
MoE expert-level specialization is a first-class interpretability primitive: architecturally monosemantic, causally validated, and controllable at inference with zero overhead.