In-Expectation Convergence of Stochastic Gradient Methods under Heavy-Tailed Noise

2026-05-30

AI Summary

This work addresses the lack of convergence guarantees for classical stochastic gradient methods under heavy-tailed noise—characterized by only finite first- or second-order moments—particularly in unbounded or non-convex settings. The authors propose a unified and concise analytical framework that, without algorithmic modifications or strong assumptions such as bounded domains, establishes for the first time expected convergence of SGD and SGDM in non-convex problems, and of SMD and ASMD in convex problems. By integrating techniques from stochastic mirror descent, momentum acceleration, and moment-condition handling under heavy-tailed distributions, the analysis overcomes key limitations in existing theory and provides a rigorous foundation for stochastic optimization in the presence of heavy-tailed noise.

Abstract

Many stochastic gradient methods are believed not to converge when the noise in stochastic gradients has only a finite $p$-th moment for $p\in\left(1,2\right)$, a setting known as the heavy-tailed noise assumption. However, some recent studies have found that Stochastic Gradient Descent ($\textsf{SGD}$), without any modification to its update rule, can surprisingly converge in expectation for convex problems with bounded domains, highlighting the potential of classical stochastic gradient methods. Inspired by this recent progress, we provide a comprehensive study of stochastic optimization under heavy-tailed noise and establish new in-expectation convergence results for Stochastic Mirror Descent ($\textsf{SMD}$) and Accelerated Stochastic Mirror Descent ($\textsf{ASMD}$) in convex optimization, and for $\textsf{SGD}$ and Stochastic Gradient Descent with Momentum ($\textsf{SGDM}$) in nonconvex optimization. Notably, our results not only hold without algorithmic changes but also avoid restrictive assumptions, such as bounded domains, imposed in prior work. More importantly, our analysis provides a new, elegant, and powerful framework for studying heavy-tailed stochastic optimization, opening a new route to understanding first-order stochastic gradient methods.

Problem

• heavy-tailed noise
• stochastic gradient methods
• in-expectation convergence
• convex optimization
• nonconvex optimization

Innovation

• heavy-tailed noise
• in-expectation convergence
• stochastic mirror descent
• nonconvex optimization
• moment conditions