Unstable Features, Reproducible Subspaces: Understanding Seed Dependence in Sparse Autoencoders

2026-06-10

AI Summary

This work addresses concerns about the reproducibility of features learned by sparse autoencoders (SAEs) across different random training seeds, which undermines their reliability for neural network interpretability. The authors propose a “feature stability” metric and conduct a systematic analysis—leveraging large-scale cross-seed experiments, automated interpretation, activation statistics, and synthetic data modeling—to characterize the functional and geometric properties of stable versus unstable features. They find that while individual unstable features are not reproducible, they collectively reside in a low-rank, reproducible subspace. Stable features dominate reconstruction and predictive signals, whereas unstable features primarily respond to low-frequency, superficial input patterns. Building on these insights, the authors develop a more stable SAE variant that significantly improves feature reproducibility without sacrificing explained variance, demonstrating that seed dependence stems from basis ambiguity rather than stochastic noise.

Abstract

Sparse autoencoders (SAEs) are widely used to interpret neural network representations, but their utility depends on whether the learned features are reproducible across training runs. We study this question through \emph{feature stability}: for each SAE feature, we estimate the probability that a similar feature reappears in an independently trained SAE. This yields a scalable per-feature signal that separates stable from unstable features. In a large-scale study across seeds, models, layers, dictionary sizes, and SAE variants, we find a pronounced functional asymmetry: stable features carry most of the reconstruction- and prediction-relevant signal, while unstable features have weak marginal impact and are dominated by low-frequency surface-form triggers in both activation statistics and automatic explanations. Geometrically, unstable features are individually non-reproducible but concentrate in reproducible lower-rank subspaces, suggesting that seed dependence often reflects basis ambiguity within a shared region of activation space rather than pure noise. A controlled synthetic model makes this mechanism explicit, showing that low-rank ground-truth features can be recovered at the subspace level while remaining non-identifiable as individual SAE latents across seeds. Finally, by pooling unique cross-seed features, we construct more stable SAEs while preserving explained variance in this setting. Together, these results show that unstable features are not merely failed or noisy latents: they have weak individual functional impact, but reflect reproducible low-dimensional structure that standard SAEs resolve differently across seeds.

Problem

• feature stability
• sparse autoencoders
• seed dependence
• reproducibility
• neural network interpretability

Innovation

• feature stability
• sparse autoencoders
• reproducible subspaces
• seed dependence
• low-rank structure