🤖 AI Summary
This study investigates the role of LayerNorm in pre-normalized recurrent Transformers, focusing on its impact on system stability and memory mechanisms. Through analytical derivations, fixed-point analysis, spectral theory, and from-scratch CPU-level training experiments across six tasks—including ablation studies—the work reveals for the first time that LayerNorm acts as an implicit gain controller within recurrent blocks. This induces a non-normal yet asymptotically contractive Jacobian, establishing that system stability is governed by spectral margin rather than operator norm. The research further clarifies that the carry term primarily stabilizes recurrent dynamics rather than encoding deep memory; genuine memory functionality arises from nonlinear recurrent pathways, with the carry term recruited for memory only under gradient descent in channel-axis-aligned tasks.
📝 Abstract
In pre-LayerNorm looped transformers, LayerNorm inside the recurrent block acts as an implicit gain controller: by coupling the block's local Lipschitz constant inversely to the activation scale, it renders the recurrence Jacobian non-normal -- asymptotically contractive at every verified fixed point even where its operator norm exceeds 1 -- so the true stability budget is the spectral margin, not an operator-norm bound. That margin depletes as the carry $ρ\to 1$, and a minority of initializations never converge to a fixed point at all, so the diagonal carry constraint $ρ(\bar{A}) < 1$ is necessary but not sufficient for convergence of the full recurrence. Training experiments across six tasks, including a controlled ablation, reveal that the linear carry is not the depth-memory mechanism: gradient descent routes memory through the block's more expressive nonlinear recurrence and leaves the stability-constrained carry at rest -- the carry's role is stabilization, not memory. We characterize the boundary of this claim: on tasks with axis-aligned per-channel structure, gradient descent does recruit the carry. All results are derived analytically and verified in a from-scratch, CPU-scale implementation; verification at larger scale is needed.