Provable Acceleration of Distributed Optimization with Local Updates

2026-01-06

arXiv.org

AI Summary

This work addresses the unresolved question of whether multiple local updates in distributed optimization can accelerate convergence under exact gradients, a setting where existing theories often rely on diminishing step sizes that may undermine potential benefits. Focusing on the classic DIGing algorithm and leveraging the Performance Estimation Problem (PEP) framework, the study provides the first rigorous proof that, over a broad class of functions, optimal acceleration is already achieved with just two local updates when an appropriate constant step size is used—additional rounds yield no further improvement. Both theoretical analysis and empirical evaluations consistently confirm this finding across synthetic and real-world datasets, offering practical guidance for designing efficient distributed optimization algorithms.

Technology Category

Search and Optimization: Distributed SearchMachine Learning: OptimizationConstraint Satisfaction and Optimization: Distributed CSP/Optimization

Application Category

Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphsResponsible Web: Human-perceived consequences of algorithmic deployment on the webSecurity and Privacy: Large-scale security measurements

Abstract

In conventional distributed optimization, each agent performs a single local update between two communication rounds with its neighbors to synchronize solutions. Inspired by the success of using multiple local updates in federated learning, incorporating local updates into distributed optimization has recently attracted increasing attention. However, unlike federated learning, where multiple local updates can accelerate learning by improving gradient estimation under mini-batch settings, it remains unclear whether similar benefits hold in distributed optimization when gradients are exact. Moreover, existing theoretical results typically require reducing the step size when multiple local updates are employed, which can entirely offset any potential benefit of these additional local updates and obscure their true impact on convergence. In this paper, we focus on the classic DIGing algorithm and leverage the tight performance bounds provided by Performance Estimation Problems (PEP) to show that incorporating local updates can indeed accelerate distributed optimization. To the best of our knowledge, this is the first rigorous demonstration of such acceleration for a broad class of objective functions. Our analysis further reveals that, under an appropriate step size, performing only two local updates is sufficient to achieve the maximal possible improvement, and that additional local updates provide no further gains. Because more updates increase computational cost, these findings offer practical guidance for efficient implementation. Extensive experiments on both synthetic and real-world datasets corroborate the theoretical findings.

Problem

• distributed optimization
• local updates
• acceleration
• exact gradients
• step size

Innovation

• distributed optimization
• local updates
• acceleration
• Performance Estimation Problems (PEP)
• DIGing algorithm