🤖 AI Summary
This work addresses the challenge of gradient staleness in asynchronous stochastic gradient descent caused by data-dependent delays, which existing methods mitigate at the cost of introducing systematic bias through discarding or downweighting delayed gradients. The paper proposes a momentum-based asynchronous optimization framework that preserves information from delayed gradients while effectively alleviating staleness. Under standard assumptions and accounting for data-dependent delays, the method establishes, for the first time, optimal convergence rates for both convex and smooth non-convex optimization problems. Furthermore, it introduces a robust adaptive learning rate scheduling strategy that substantially simplifies hyperparameter tuning. Together, the theoretical analysis and algorithmic design offer a novel analytical perspective and practical tools for asynchronous optimization.
📝 Abstract
Asynchronous stochastic gradient descent (SGD) enables scalable distributed training but suffers from gradient staleness. Existing mitigation strategies, such as delay-adaptive learning rates and staleness-aware filtering, typically attenuate or discard delayed gradients, introducing systematic bias: updates from simpler or faster-to-process samples are overrepresented, while gradients from more complex samples are delayed or suppressed. In contrast, prior approaches to data-dependent delays rely on a Lipschitz assumption that yields suboptimal rates or leave the smooth, convex case unaddressed. We propose a momentum-based asynchronous framework designed to preserve information from delayed gradients while mitigating the effects of staleness. We establish the first optimal convergence rates for data-dependent delays in both convex and non-convex smooth setups, providing a new result for asynchronous optimization under standard assumptions. Additionally, we derive robust learning-rate schedules that simplify hyperparameter tuning in practice.