Mechanisms of Width Scaling in Normalized Residual Networks: The Effective Alignment Dimension

2026-07-27

AI Summary

This work addresses the challenge that existing neural network width theories fail to guarantee the generalization of widening directions identified during training. Focusing on function-preserving residual expansions, the study investigates the alignment between training and test gradients, introducing the notion of “effective alignment dimension” to characterize the signal-to-noise geometric structure of activation gradients. For the first time, this measurable quantity enables a high-probability guarantee of improved test risk under finite samples—without requiring assumptions on covariance spectra or predetermined width growth rates. The theoretical analysis leverages mean-variance decomposition of inner products of activation gradients within a residual expansion framework. Experiments on LLaMA-style Transformers, Pythia, and ResNet-20 demonstrate that wider models exhibit higher effective alignment dimensions and lower empirical misalignment, with this metric accurately predicting both the direction and magnitude of held-out loss changes.

Abstract

Existing theories of neural-network width characterize asymptotic limits, but provide limited guidance on whether an expansion direction identified from finite training data remains beneficial on unseen data. We study this problem for function-preserving residual expansion and introduce the effective alignment dimension, a measurable quantity describing the signal-noise geometry of activation gradients. By deriving the exact mean and variance of the inner product between independently estimated training and test gradients, we obtain a finite-sample upper bound on misalignment probability. The bound depends only on the effective alignment dimension and an effective sample size, requiring finite second moments and a nonzero population gradient, without covariance spectral assumptions or prescribed width-growth rates. We integrate this certificate into the train-test residual-expansion framework, yielding a high-probability condition for test-risk improvement. Experiments across width-controlled LLaMA-style Transformers, Pythia, and ResNet-20 show that wider models exhibit larger effective alignment dimensions and lower empirical misalignment. Direct residual interventions confirm that the alignment statistic predicts the sign and magnitude of held-out loss changes.

Problem

• width scaling
• residual networks
• train-test alignment
• effective alignment dimension
• generalization

Innovation

• effective alignment dimension
• residual expansion
• finite-sample analysis
• gradient alignment
• width scaling