Second-Moment Stochastic Approximation Methods

2026-09-28

AI Summary

This study addresses the efficiency limitations of optimizers such as Adam, which arise from traditional stochastic approximation relying solely on first-order moments. For the first time, second-moment estimation is incorporated into the stochastic approximation framework, enabling a unified derivation of both the Adam and Muon algorithms from an optimal preconditioning perspective alongside a two-stage convergence analysis framework. Leveraging Dvoretzky's theorem, this work rigorously establishes almost sure convergence to a neighborhood of the optimum, thereby overcoming the theoretical constraints inherent in single-moment approaches. Furthermore, it develops a general theory encompassing mainstream deep learning optimizers and provides explicit bounds on the convergence radii for Muon and spectral variants of Adam, substantially deepening the theoretical understanding of optimization mechanisms.

Abstract

Classical stochastic approximation methods rely on estimators of the first moment (mean) of a random regression function. We study methods that employ estimators of both the first and the second moments, which include modern deep-learning optimizers such as Adam and Muon as special cases. We derive second-moment stochastic approximation methods through the lens of optimal preconditioning for solving matrix equations, and develop a two-stage framework for their convergence analysis. The first stage focuses on the analysis of conceptual (impractical) methods that rely on the exact first and second moments. In the second stage, we replace the exact moments with their respective estimators, and invoke Dvoretzky's theorem to show that the resulting practical methods converge almost surely to a neighborhood of the target solution. The size of the neighborhood depends on the biases and variances of the first- and second-moment estimators. We derive concrete bounds for Muon and a spectral variant of Adam that determine the radius of their neighborhood of convergence.

Problem

• Stochastic Approximation
• Second-Moment Estimation
• Deep Learning Optimizers
• Convergence Analysis
• Optimal Preconditioning

Innovation

• Second-Moment Stochastic Approximation
• Optimal Preconditioning
• Two-Stage Convergence Analysis
• Dvoretzky's Theorem
• Deep Learning Optimizers