Convergence Analysis of STORM Under Different Geometries

2026-10-01

AI Summary

This work addresses the limitation of the STORM algorithm in non-convex optimization, which relies on a strong average smoothness assumption, by proposing an improved framework based on a unified recursive structure. By constructing auxiliary sequences to contrast update processes and integrating stochastic recursive momentum with variance reduction techniques, we establish convergence analysis under standard smoothness conditions. Theoretically, we demonstrate that the proposed method adapts to multiple objective classes solely through hyperparameter tuning: it achieves optimal convergence rates of O(T^{-1/4}), O(σR/√T), and O(σ²/(λT)) for non-convex, convex, and strongly convex objectives satisfying the Polyak–Łojasiewicz (PL) condition, respectively. This successfully eliminates the average smoothness assumption while verifying the algorithm's versatility across diverse optimization landscapes.

Abstract

Stochastic recursive momentum (STORM) achieves fast convergence for nonconvex optimization via the variance reduction effect, but existing analyses rely on the strong average smoothness assumption. In this paper, we study the convergence of STORM for different objectives without average smoothness. We first revisit the results under average smoothness, obtaining the $O(T^{-1/3})$ bound for nonconvex objectives and the $O(σ^2/(μT))$ bound for last-iterate output under the $μ$-Polyak--Łojasiewicz~(PL) condition. Without average smoothness, we design an auxiliary sequence and compare the STORM update with it in the analysis. With the help of this sequence, we prove that STORM still attains an $O(T^{-1/4})$ rate for nonconvex objectives, which is optimal under standard smoothness. For convex and $λ$-strongly convex objectives, we further prove averaged and last-iterate bounds with optimal rates of $O(σR/\sqrt T)$ and $O(σ^2/(λT))$, respectively. All the obtained results use the same STORM recursion with different hyperparameter choices.

Problem

• STORM
• convergence analysis
• nonconvex optimization
• average smoothness
• variance reduction

Innovation

• STORM
• variance reduction
• convergence analysis
• auxiliary sequence
• nonconvex optimization