Emergent Inverse-Depth Scaling From Nonlinearity In Attention

2026-10-07

AI Summary

This study addresses the unclear role of nonlinear attention in governing loss evolution with depth within neural scaling laws. By integrating theoretical analysis with empirical validation, the authors leverage data spectral analysis and the central limit theorem to derive the scaling behavior of model loss as a function of depth. This work provides the first proof that nonlinear attention induces a universal inverse-depth decay phenomenon, challenging conventional linear theories predicated on global spectral strength. Furthermore, it reveals that the local focusing mechanism enables models to selectively attend to relevant tokens, facilitating the parallel learning of strong and weak feature directions. These findings offer a novel perspective for understanding depth-induced gains in large language models.

Abstract

Scaling laws describe power-law improvements in model performance with dataset size and parameter count, yet their underlying mechanisms are not fully understood. To explain the parameter count scaling, existing theory posits power-law scaling with model depth. In linear-attention models, this scaling is tied to a power-law data spectrum: unable to selectively attend to relevant tokens, these models learn according to global spectral strength, with stronger directions learned before weaker ones. Large language models, however, can be strongly nonlinear. Here, we show that nonlinear attention yields inverse-depth decay of loss across all tested data spectra. Nonlinearity enables attention to focus selectively on relevant tokens, allowing strong and weak spectral directions to be learned in parallel. Similar focusing across layers motivates a connection to the central limit theorem: shared error across layers sets the loss plateau, while aggregation turns layer-specific differences into continued gains with depth. Our findings suggest that depth scaling may arise from nonlinearity in attention, which allows large language models to focus locally and may make the global covariance structure less relevant.

Problem

• scaling laws
• nonlinear attention
• depth scaling
• large language models
• inverse-depth decay

Innovation

• nonlinear attention
• inverse-depth scaling
• scaling laws
• selective focusing
• central limit theorem