The Tell-Tale Norm: $\ell_2$ Magnitude as a Signal for Reasoning Dynamics in Large Language Models

2026-06-04

AI Summary

This work addresses the absence of reliable, endogenous signals that capture layer-wise dynamics within the reasoning processes of large language models. The authors discover that the ℓ2 norm of hidden states effectively characterizes a model’s reasoning intensity and, for the first time, establish a theoretical link between this norm and feature activations in sparse autoencoders (SAEs). Building on this insight, they propose three test-time inference enhancement techniques that require no additional training, leveraging the ℓ2 norm to guide adaptive reasoning strategies. The approach is validated through causal intervention and correlation analysis. Experiments across diverse model architectures and reasoning benchmarks demonstrate consistent and significant performance improvements, confirming the ℓ2 norm as a valid endogenous indicator of reasoning dynamics.

Abstract

Recent work has sought to understand Large Language Models (LLMs) reasoning, yet a principled, model-intrinsic signal that captures its layer-wise reasoning dynamics remains underexplored. We bridge this gap by demonstrating that the l2 norm of hidden states serves as an endogenous signal of the model's reasoning intensity. Using Sparse Autoencoders (SAEs) as a diagnostic probe, we observe that LLMs' internal reasoning is marked by a sharp increase in reasoning feature activations concentrated in late layers. Motivated by this pattern, we establish a formal link between reasoning intensity and the model's latent geometry and theoretically prove that the l2 norm of hidden states bounds the activation strength of SAE reasoning features. Empirical correlation analysis and causal interventions further validate the l2 norm as a faithful indicator, where heightened norms consistently correspond to critical reasoning steps. We then introduce three test-time scaling techniques guided by l2 norms: (i) Adaptive Layer-wise Reasoning Recursion, (ii) Endogenous Reasoning State Steering, and (iii) l2-guided Response Selection, which requires no additional training or data and is compatible with advanced inference engines. Experiments across model architectures and benchmarks show that l2-norm-based techniques significantly improve reasoning performance, offering a principled yet simple lens to perceive and control LLM latent reasoning dynamics. Our code is available at https://github.com/zjy1298/The-Tell-Tale-Norm.

Problem

• reasoning dynamics
• large language models
• hidden states
• l2 norm
• model-intrinsic signal

Innovation

• l2 norm
• reasoning dynamics
• Sparse Autoencoders
• test-time scaling
• latent geometry