Tight Time Complexities in Parallel Stochastic Optimization with Arbitrary Computation Dynamics

2024-08-09

arXiv.org

AI Summary

Distributed stochastic optimization faces arbitrary computational dynamics—including hardware disconnections, time-varying compute capacity, and fluctuating processing speeds—rendering existing models inadequate for real-world deployment. Method: We propose the first general asynchronous computation model encompassing all realistic scenarios; based on it, we derive tight time-complexity lower bounds (up to constant factors) for mainstream synchronous and asynchronous methods—including Minibatch SGD, Async SGD, and Picky SGD—and prove that Rennala/Malenia SGD achieves optimal convergence. Our analysis integrates general computational dynamical modeling, information-theoretic lower-bound derivation, and stochastic optimization convergence theory. Contribution: We establish fundamental theoretical limits and design principles for system-aware optimization, providing foundational support for fault-tolerant, robust distributed learning. The results unify treatment of heterogeneous, unreliable, and dynamic execution environments while delivering precise complexity characterizations grounded in both system behavior and statistical learning theory.

Technology Category

• Reasoning under Uncertainty: Stochastic Optimization
• Machine Learning: Optimization
• Search and Optimization: Learning to Search

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems
• Security and Privacy: Large-scale security measurements

Abstract

In distributed stochastic optimization, where parallel and asynchronous methods are employed, we establish optimal time complexities under virtually any computation behavior of workers/devices/CPUs/GPUs, capturing potential disconnections due to hardware and network delays, time-varying computation powers, and any possible fluctuations and trends of computation speeds. These real-world scenarios are formalized by our new universal computation model. Leveraging this model and new proof techniques, we discover tight lower bounds that apply to virtually all synchronous and asynchronous methods, including Minibatch SGD, Asynchronous SGD (Recht et al., 2011), and Picky SGD (Cohen et al., 2021). We show that these lower bounds, up to constant factors, are matched by the optimal Rennala SGD and Malenia SGD methods (Tyurin&Richt'arik, 2023).

Problem

• Establishes optimal time complexities in distributed stochastic optimization
• Addresses hardware and network delays in parallel computation
• Proves tight lower bounds for synchronous and asynchronous methods

Innovation

• Universal computation model
• Tight lower bounds
• Optimal Rennala SGD