On the Provable Suboptimality of Momentum SGD in Nonstationary Stochastic Optimization

2026-01-18

AI Summary

This work investigates the degradation of tracking performance in momentum-based stochastic gradient descent (SGD) under non-stationary stochastic optimization, where both data distributions and optimal parameters drift over time. By developing an error decomposition framework that accounts for initialization, noise, and drift components, and leveraging dynamic regret analysis, gradient variation constraints, and strong convexity with smoothness assumptions, the study establishes—for the first time—theoretical evidence that momentum introduces an irreducible “inertial window” in drift-dominated regimes, constituting an information-theoretic performance bottleneck. The authors derive a minimax lower bound on dynamic regret, demonstrating that momentum significantly amplifies tracking error induced by parameter drift, and that this error becomes unbounded as the momentum parameter approaches one. These results rigorously delineate the theoretical conditions under which standard SGD outperforms its momentum variants.

Technology Category

• Reasoning under Uncertainty: Stochastic Optimization
• Machine Learning: Optimization
• Search and Optimization: Non-convex Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Web Mining and Content Analysis: Content-based information diffusion
• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience

Abstract

In this paper, we provide a comprehensive theoretical analysis of Stochastic Gradient Descent (SGD) and its momentum variants (Polyak Heavy-Ball and Nesterov) for tracking time-varying optima under strong convexity and smoothness. Our finite-time bounds reveal a sharp decomposition of tracking error into transient, noise-induced, and drift-induced components. This decomposition exposes a fundamental trade-off: while momentum is often used as a gradient-smoothing heuristic, under distribution shift it incurs an explicit drift-amplification penalty that diverges as the momentum parameter $\beta$ approaches 1, yielding systematic tracking lag. We complement these upper bounds with minimax lower bounds under gradient-variation constraints, proving this momentum-induced tracking penalty is not an analytical artifact but an information-theoretic barrier: in drift-dominated regimes, momentum is unavoidably worse because stale-gradient averaging forces systematic lag. Our results provide theoretical grounding for the empirical instability of momentum in dynamic settings and precisely delineate regime boundaries where vanilla SGD provably outperforms its accelerated counterparts.

Problem

• nonstationary optimization
• momentum SGD
• tracking error
• dynamic regret
• stochastic optimization

Innovation

• momentum SGD
• nonstationary optimization
• tracking error
• dynamic regret
• inertia penalty