🤖 AI Summary
This work investigates the dynamic emergence of linear conceptual representations—referred to as “abstractions”—during neural network training, rather than merely their static presence at convergence. By developing a theoretical framework, the authors derive the first closed-form solution for abstraction trajectories in linear networks and extend this analysis to nonlinear architectures with ReLU and erf activations, integrating geometric reasoning with generalization-aware linear probing. The study uncovers three core principles: abstractions are jointly shaped by data and target geometry, amplified with network depth, and modulated by initialization scale. It further identifies an abstraction attenuation phenomenon in nonlinear networks, where post-activation features exhibit weaker abstraction than pre-activation ones. These theoretical predictions are validated on open-source models such as DINOv3 and Gemma, and leveraged to enhance the generalization performance of linear probes in large language models.
📝 Abstract
In artificial and biological neural networks, concepts are often encoded as consistent linear directions in representation space. In deep learning, this idea is known as the linear representation hypothesis and underpins many interpretability and control methods based on linear probes, from concept detection to activation steering. Yet while prior work has studied whether such directions should exist $\textit{after}$ training, the dynamics of how they emerge $\textit{during}$ training remain poorly understood. Here, we develop a framework to study the alignment of concept directions during training - a process we call "abstraction". In a minimal linear network setting, we obtain exact solutions for the full trajectory of abstraction. These solutions reveal key analytic principles governing abstraction: (i) data and target geometry jointly determine abstraction at the end-of-learning, (ii) abstraction improves with network depth, and (iii) initialization scale controls the maximum abstraction reached during training. Extending our theory to nonlinear networks, we analyze how the choice of nonlinearity affects abstraction dynamics: erf networks approximate the linear theory, while abstraction in ReLU networks depends less on target geometry and more on input geometry. Across both, we prove a striking attenuation law: both nonlinearities weaken abstraction in activations relative to preactivations. We find evidence for this law in open models (DINOv3, Gemma 4) and apply our theory to improve linear probe generalization in LLMs. Together, our results provide a dynamical theory of abstraction with implications for interpretability and control.