Gaussian Approximation and Multiplier Bootstrap for Stochastic Gradient Descent

2025-02-10

AI Summary

This paper establishes non-asymptotic convergence rates for the central limit theorem (CLT) of Polyak–Ruppert averaged stochastic gradient descent (SGD) iterates and proves the non-asymptotic validity of the multiplier bootstrap for constructing confidence sets around the optimal solution. Methodologically, it achieves Gaussian approximation without estimating the limiting covariance matrix—yielding a convex distance convergence rate of $O(1/sqrt{n})$—by integrating optimization iteration analysis with bootstrap design within the nonlinear statistic Gaussian approximation framework of Shao & Zhang (2022). The key contribution is the first rigorous demonstration of non-asymptotic consistency of the multiplier bootstrap for averaged SGD, thereby circumventing the traditional reliance on covariance estimation in asymptotic inference. This yields a provably valid, hyperparameter-free, and computationally tractable uncertainty quantification tool for machine learning model parameters.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Stochastic Optimization
• Search and Optimization: Non-convex Optimization

Application Category

• Security and Privacy: Large-scale security measurements
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

In this paper, we establish non-asymptotic convergence rates in the central limit theorem for Polyak-Ruppert-averaged iterates of stochastic gradient descent (SGD). Our analysis builds on the result of the Gaussian approximation for nonlinear statistics of independent random variables of Shao and Zhang (2022). Using this result, we prove the non-asymptotic validity of the multiplier bootstrap for constructing the confidence sets for the optimal solution of an optimization problem. In particular, our approach avoids the need to approximate the limiting covariance of Polyak-Ruppert SGD iterates, which allows us to derive approximation rates in convex distance of order up to $1/sqrt{n}$.

Problem

• non-asymptotic convergence rates
• Polyak-Ruppert-averaged iterates
• multiplier bootstrap validity

Innovation

• Gaussian Approximation for SGD
• Multiplier Bootstrap Confidence Sets
• Avoids Limiting Covariance Approximation