Sparsely gated tiny linear experts

2026-06-05

AI Summary

Although Mixture-of-Experts (MoE) models exhibit global sparsity, individual experts remain large and dense, limiting both computational efficiency and interpretability. This work proposes replacing each expert with a single linear neuron and integrating an extremely sparse gating mechanism to construct a highly sparse feedforward layer. By eliminating nonlinear activations, the design achieves native interpretability and enhanced computational efficiency. Under isoflop conditions—matching total computational cost—the model consistently reduces language modeling perplexity across varying compute budgets. Furthermore, the learned feedforward neurons exhibit clear semantic clustering, enabling direct causal analysis for factual recall without additional probing or fine-tuning.

Abstract

Sparsity allows scaling model parameters without proportionally increasing computational cost. While mixture of experts (MoE) models are made increasingly sparse, individual experts typically remain large and dense. Here, we demonstrate that further increasing sparsity by shrinking each expert to consist of a single neuron and selecting a tiny fraction of many available neurons can improve compute efficiency and interpretability. Counterintuitively, the key to achieving both is removing the nonlinearity typically applied to the experts, resulting in a network of sparsely gated linear neurons (sgatlin). In an isoflop comparison, we find that replacing all transformer feedforward layers with sgatlin improves perplexity in language models across different compute budgets. At the same time, the sparsity and linearity of the resulting feedforward circuits present new opportunities for model interpretability. In a small-scale case study, we demonstrate that feedforward circuits in sgatlin can be interpreted without having to train additional replacement models. We find that they form semantically structured clusters and are causally implicated in factual recall. Our findings paint a possible path towards compute-efficient and interpretable transformer feedforward layers.

Problem

• sparsity
• mixture of experts
• interpretability
• compute efficiency
• linear experts

Innovation

• sparsely gated linear neurons
• mixture of experts
• model interpretability
• compute efficiency
• linear experts