🤖 AI Summary
In heterogeneous data and system environments, asynchronous stochastic gradient descent (ASGD) biases optimization toward a frequency-weighted average of local objectives due to faster workers updating more frequently, thereby deviating from the true global optimum. This work proposes scaling each worker’s learning rate inversely proportional to its computation time, ensuring that all workers contribute equal total learning rate per unit time—without altering the standard ASGD mechanism or requiring synchronization, buffering, or additional memory. Theoretically, this approach achieves, for the first time in a fixed-computation model under non-convex settings, unbiased convergence to the correct global objective, with the leading term matching the known lower bound on time complexity; asynchronous delays and data heterogeneity affect only lower-order terms. Experiments confirm convergence to the correct solution and demonstrate performance competitive with or superior to state-of-the-art methods.
📝 Abstract
Asynchronous stochastic gradient descent (ASGD) is a standard way to exploit heterogeneous compute resources in distributed learning: instead of forcing fast workers to wait for slow ones, the server updates the model whenever a gradient arrives. Vanilla ASGD applies each arriving gradient with the same weight. When local data distributions are heterogeneous, this becomes problematic: faster workers contribute more updates, and we show theoretically that the method is biased toward a frequency-weighted average of the local objectives rather than the desired global objective. Existing remedies typically move away from the simple ASGD template by introducing gathering phases, buffering, or extra memory. We show that this is unnecessary. Keeping the standard ASGD mechanism, we recover the correct objective by rescaling worker-specific stepsizes in proportion to their computation times, so that each worker contributes the same aggregate learning rate over a cycle. In the non-convex setting, under smoothness and bounded heterogeneity assumptions, we prove that the resulting method, Rescaled ASGD, converges to stationary points of the correct global objective in the fixed-computation model. Its time complexity matches the known lower bound in the leading term, while the effects of staleness and data heterogeneity appear only in lower-order terms. Experiments confirm that the method converges to the correct objective and is competitive with state-of-the-art baselines.