Darkness Visible: Reading the Exception Handler of a Language Model

2026-04-06

AI Summary

This study investigates how language models implement anomaly handling and knowledge routing through neurons rather than directly storing factual knowledge. Focusing on the final-layer MLP of GPT-2 Small, the authors combine neuron decomposition, residual stream analysis, targeted interventions, and statistical validation to identify and formally define a structured three-tier anomaly-processing mechanism. They systematically categorize all 3,072 neurons into four functional types: Core, Differentiator, Specialist, and Consensus. The work demonstrates that MLP modules merely amplify or suppress signals already generated by attention, with routing functionality emerging only in the terminal layer, revealing that so-called “knowledge neurons” primarily serve as routing infrastructure. Additionally, the study uncovers an interaction between consensus and anomaly pathways and a reversed garden-path effect, elucidating the cooperative principles underlying neuronal contributions to contextual prediction.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Natural Language Processing: (Large) Language Models
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

The final MLP of GPT-2 Small exhibits a fully legible routing program -- 27 named neurons organized into a three-tier exception handler -- while the knowledge it routes remains entangled across ~3,040 residual neurons. We decompose all 3,072 neurons (to numerical precision) into: 5 fused Core neurons that reset vocabulary toward function words, 10 Differentiators that suppress wrong candidates, 5 Specialists that detect structural boundaries, and 7 Consensus neurons that each monitor a distinct linguistic dimension. The consensus-exception crossover -- where MLP intervention shifts from helpful to harmful -- is statistically sharp (bootstrap 95% CIs exclude zero at all consensus levels; crossover between 4/7 and 5/7). Three experiments show that "knowledge neurons" (Dai et al., 2022), at L11 of this model, function as routing infrastructure rather than fact storage: the MLP amplifies or suppresses signals already present in the residual stream from attention, scaling with contextual constraint. A garden-path experiment reveals a reversed garden-path effect -- GPT-2 uses verb subcategorization immediately, consistent with the exception handler operating at token-level predictability rather than syntactic structure. This architecture crystallizes only at the terminal layer -- in deeper models, we predict equivalent structure at the final layer, not at layer 11. Code and data: https://github.com/pbalogh/transparent-gpt2

Problem

• exception handler
• knowledge representation
• neuron interpretability
• language model
• routing mechanism

Innovation

• exception handler
• neuron decomposition
• routing mechanism
• knowledge neuron
• garden-path effect