Are Single-Token Sparse Autoencoder Features Causally Necessary? Layer-Depth and SAE-Family Effects

2026-07-22

AI Summary

This study investigates the causal necessity, stability, and interpretability foundations of single-token sparse autoencoder (SAE) features in language models. Conducting full-layer causal analysis across 3.9 million features in six models and three SAE families, the work employs zero-ablation and Benjamini–Hochberg significance testing. It provides the first systematic evidence that the causal effects of SAE features are primarily governed by training methodology rather than activation functions or model scale, reveals that layer depth dictates damage propagation patterns, and uncovers substantial differences in causal behavior across SAE families. Results show significant logit degradation in 178 out of 208 layers; single-token features exhibit 4.7× tighter clustering in decoder space, predominantly in early layers, with target token rankings recoverable to within twice the baseline for the top 96–98%.

Abstract

Sparse autoencoder (SAE) features are used to interpret and steer large language models, yet whether a feature's causal role is stable across SAE families remains untested. Single-token features that activate on one vocabulary item provide the diagnostic case where ground truth permits direct comparison. We analyze 3.9M features across six models and three SAE families using zero-ablation at full layer depth. Single-token features cluster 4.7x tighter in decoder space and concentrate in early layers (Layer 0 in GPT2-Small; L0-L4 in Gemma). Ablating them yields Benjamini-Hochberg-significant logit reductions in 178 of 208 full-layer conditions, with depth controlling whether damage cascades downstream or shapes the output directly. Cross-family causal differences exceed within-family scale effects: on the same base model, GemmaScope and BatchTopK features remain causally anchored, while LlamaScope features are locally redundant. The target token's rank recovers to within 2x baseline 96-98% of the time after the same ablation, and a controlled activation-function comparison reverses sign within the same model, leaving training recipe as the residual candidate. Cross-family interpretability claims are therefore sensitive to training methodology, not just activation function or scale.

Problem

• sparse autoencoder
• causal necessity
• single-token features
• interpretability
• training methodology

Innovation

• sparse autoencoder
• causal necessity
• zero-ablation
• cross-family comparison
• feature interpretability