Convergence Analysis of Adaptive Gradient Methods under Refined Smoothness and Noise Assumptions

2024-06-07

arXiv.org

AI Summary

This paper addresses the theoretical advantages of adaptive methods—specifically AdaGrad—in stochastic nonconvex optimization. Departing from conventional assumptions of global Lipschitz continuity and uniform noise variance, it introduces coordinate-wise fine-grained smoothness and heterogeneous noise variance conditions. Crucially, it adopts the ℓ₁-norm to measure stationarity—i.e., proximity to a gradient stationary point—for the first time in this context. Theoretically, it establishes that AdaGrad achieves an iteration complexity of O(1/ε²), strictly improving upon SGD’s O(d/ε²) and yielding a d-fold acceleration. An information-theoretic lower bound is constructed to confirm that this upper bound is tight up to logarithmic factors. Furthermore, the work develops a novel convergence analysis framework based on the ℓ₁-norm, uncovering the intrinsic advantage of adaptive step sizes in handling coordinate-wise heterogeneity of problem structure and noise.

Technology Category

• Search and Optimization: Non-convex Optimization
• Reasoning under Uncertainty: Stochastic Optimization
• Machine Learning: Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems

Abstract

Adaptive gradient methods, such as AdaGrad, are among the most successful optimization algorithms for neural network training. While these methods are known to achieve better dimensional dependence than stochastic gradient descent (SGD) under favorable geometry for stochastic convex optimization, the theoretical justification for their success in stochastic non-convex optimization remains elusive. In fact, under standard assumptions of Lipschitz gradients and bounded noise variance, it is known that SGD is worst-case optimal (up to absolute constants) in terms of finding a near-stationary point with respect to the $ell_2$-norm, making further improvements impossible. Motivated by this limitation, we introduce refined assumptions on the smoothness structure of the objective and the gradient noise variance, which better suit the coordinate-wise nature of adaptive gradient methods. Moreover, we adopt the $ell_1$-norm of the gradient as the stationarity measure, as opposed to the standard $ell_2$-norm, to align with the coordinate-wise analysis and obtain tighter convergence guarantees for AdaGrad. Under these new assumptions and the $ell_1$-norm stationarity measure, we establish an upper bound on the convergence rate of AdaGrad and a corresponding lower bound for SGD. In particular, for certain configurations of problem parameters, we show that the iteration complexity of AdaGrad outperforms SGD by a factor of $d$. To the best of our knowledge, this is the first result to demonstrate a provable gain of adaptive gradient methods over SGD in a non-convex setting. We also present supporting lower bounds, including one specific to AdaGrad and one applicable to general deterministic first-order methods, showing that our upper bound for AdaGrad is tight and unimprovable up to a logarithmic factor under certain conditions.

Problem

• Proving AdaGrad's complexity advantage over SGD in non-convex optimization
• Refining assumptions for adaptive gradient methods' coordinate-wise analysis
• Establishing tight convergence bounds for AdaGrad under l1-norm stationarity

Innovation

• Refined smoothness and noise assumptions
• Adopts l1-norm for stationarity measure
• Proves AdaGrad's complexity gain over SGD