Gaussian Limits for SGD Without Stationary Moments

2026-10-04

AI Summary

This study addresses a fundamental contradiction in the Gaussian approximation of stochastic gradient descent (SGD), where temporal dependence causes divergence of higher-order moments of the stationary error while the limiting distribution remains Gaussian, thereby rendering traditional moment analysis ineffective. To resolve this, we establish a general theoretical framework that derives the Ornstein–Uhlenbeck process limit via path contraction, score cancellation, and localization techniques, and further analyzes exact conditional Gaussian laws alongside covariance corrections under independent observations. Our contribution transcends the limitations of moment-based analysis by establishing precise probabilistic approximations and first-order total variation constants even when higher-order moments diverge. Empirical results demonstrate that incorporating dependency-aware scoring yields significant improvements in distributional error, coverage rates, and calibration performance.

Abstract

Temporal dependence can separate the Gaussian approximation of stochastic gradient descent from its stationary moments. For unmodified least-squares SGD, we construct a design with standard Gaussian marginals whose stationary error has every positive moment infinite. Independent observations with the same marginals instead give finite stationary variance. Both regimes retain a Gaussian small-step limit. Our general theory establishes pathwise contraction from a finite second design moment, then uses score cancellation and localization to obtain stationary Gaussian and Ornstein--Uhlenbeck limits. Independent Gaussian regression errors yield an exact conditional Gaussian law and total-variation convergence under the same design integrability. Stronger design conditions identify a positive first-order total-variation constant and a deterministic covariance correction with $o(a)$ error. A scalar coverage expansion translates this correction into its inference consequence. Experiments examine distributional error, coverage, and calibration with dependent scores. Together, these results establish precise probability-law approximation beyond moment-based stationary analysis.

Problem

• Stochastic Gradient Descent
• Gaussian Limits
• Stationary Moments
• Temporal Dependence
• Probability-law Approximation

Innovation

• Stochastic Gradient Descent
• Gaussian Limits
• Ornstein-Uhlenbeck Process
• Total-Variation Convergence
• Score Cancellation