A Geometric View for Understanding Concept Learning and Neuron Interpretation in Sparse Autoencoders

2026-06-05

AI Summary

This work addresses the lack of rigorous mathematical definitions for “concepts” and “learning” in existing sparse autoencoders, which obscures the mechanisms underlying neuron interpretability. The authors formalize concepts as sets of data points and frame concept learning as a set-alignment problem between human-defined concepts and model-induced concepts, distinguishing three hierarchical levels: detection, separation, and approximation. Building on geometric and set-theoretic foundations, they propose a unified theoretical framework that elucidates the origins of phenomena such as feature splitting, absorption, familial relationships, and hierarchical structure. Leveraging formal concept analysis, the framework captures the many-to-many correspondence between neurons and concepts. Theoretical predictions are validated on synthetic data, revealing how model scale and sparsity jointly influence concept learning capacity, and enabling the construction of concept lattices that systematically organize neuron–concept mappings.

Abstract

We propose a unified mathematical framework for a geometric understanding of concept learning and neuron interpretation in sparse autoencoders (SAEs). While SAEs improve interpretability of neural networks by learning sparse feature representations, a principled definition of ''concept'' and ''learning'' remains unclear. We formalize concepts as sets of data points and cast concept learning as a set-alignment problem between human-defined and model-induced concepts. This formulation distinguishes three increasingly strong notions of learning -- detection, separation, and approximation -- and yields geometric conditions, error bounds, and capacity constraints for when concepts can be represented by individual neurons or multi-neuron units. It also provides a set-theoretic account for common SAE phenomena, including feature splitting, feature absorption, feature families, and hierarchical concepts. Finally, we connect concept learning and neuron interpretation through formal concept analysis, showing that the two directions need not agree and that their many-to-many structure can be organized by concept lattices. Experiments on synthetic data with ReLU and Top-$K$ SAEs illustrate the theory and reveal the effects of SAE size and sparsity on concept learning.

Problem

• concept learning
• neuron interpretation
• sparse autoencoders
• geometric understanding
• feature representation

Innovation

• concept learning
• neuron interpretation
• sparse autoencoders
• geometric framework
• formal concept analysis