🤖 AI Summary
This study investigates how weight decay enhances training stability in deep learning through a unified framework combining dynamical systems analysis, the Edge of Stability (EoS) theory, the Neural Tangent Kernel (NTK) perspective, and mathematical modeling. The authors demonstrate that weight decay induces architecture-dependent phase transitions in both CNNs and MLPs, rooted in the global alignment between parameter vectors and curvature gradients. This mechanism effectively suppresses asymptotic sharpness and modulates oscillations in optimization trajectories. Furthermore, the work reveals that conventional curvature-based thresholds derived under convexity assumptions fail under regularization, thereby establishing weight decay as a nontrivial yet essential regulator of stable training dynamics.
📝 Abstract
In modern deep learning, weight decay is often credited with "stabilizing" training dynamics, diverging from its classical role as a static regularization penalty. We investigate a fundamental question: *does weight decay stabilize training dynamics, and if so, through which mechanism?* Indeed, training stability is understood through different but related notions in the literature. We consider how weight decay affects the parameter-space dynamics and loss sharpness by analyzing its effects at the \emph{Edge of Stability} (EoS). We show that weight decay robustly slows *progressive sharpening}. Furthermore, we uncover a striking architecture-dependent phase transition. In CNNs, weight decay dampens the oscillations at the EoS, while in MLPs, increasing weight decay causes a phase transition in which the sharpness stabilizes at a threshold significantly below the theoretical $\frac{2}η$ boundary. We develop a mathematical framework that accurately models these phenomena and identify the global alignment of the parameter vector and the sharpness gradient as the mechanistic driver of the phase transition. Importantly, we show that these phenomena translate into stability in terms of search in function-space (NTK). Last, this shows that curvature thresholds obtained from convex/quadratic heuristics may not be reliable stability diagnostics under regularization.