🤖 AI Summary
This work investigates the fundamental performance limits of distributed stochastic optimization under communication compression. We establish the first set of tight convergence lower bounds—covering six distinct settings formed by combining strongly convex, convex, and nonconvex objective functions with unbiased and contractive compressors. Building upon these bounds, we propose NEOLITHIC, the first compression-based algorithm achieving near-optimal rates (up to logarithmic factors) across all six settings. NEOLITHIC integrates variance reduction, momentum acceleration, and explicit compressor modeling. We prove theoretically that it matches the derived lower bounds under mild assumptions. Empirical evaluations demonstrate that, in multi-node training, NEOLITHIC reduces communication overhead by 3–5× compared to state-of-the-art compressed methods, while significantly improving convergence efficiency.
📝 Abstract
Communication compression is an essential strategy for alleviating communication overhead by reducing the volume of information exchanged between computing nodes in large-scale distributed stochastic optimization. Although numerous algorithms with convergence guarantees have been obtained, the optimal performance limit under communication compression remains unclear. In this paper, we investigate the performance limit of distributed stochastic optimization algorithms employing communication compression. We focus on two main types of compressors, unbiased and contractive, and address the best-possible convergence rates one can obtain with these compressors. We establish the lower bounds for the convergence rates of distributed stochastic optimization in six different settings, combining strongly-convex, generally-convex, or non-convex functions with unbiased or contractive compressor types. To bridge the gap between lower bounds and existing algorithms' rates, we propose NEOLITHIC, a nearly optimal algorithm with compression that achieves the established lower bounds up to logarithmic factors under mild conditions. Extensive experimental results support our theoretical findings. This work provides insights into the theoretical limitations of existing compressors and motivates further research into fundamentally new compressor properties.
