Controlled Periodic Synchronization for Efficient Data-Parallel Training

2026-07-23

AI Summary

This work addresses the communication latency and network sensitivity caused by frequent gradient synchronization in geographically distributed data-parallel training. The authors propose CPDP, a novel strategy that explicitly models synchronization frequency as a tunable system parameter and introduces a hybrid coordination mechanism combining gradient AllReduce with SlowMo-style parameter averaging. This approach significantly enhances communication efficiency while preserving convergence guarantees. Implemented within a PyTorch DDP-compatible framework, CPDP achieves a 2.44 percentage point improvement in test accuracy over standard DDP on ResNet-50 trained on CIFAR-100 (with K=4), reduces average training time by 13.8%, and cuts synchronization exposure time by approximately 50%. Consistent gains are also demonstrated on ViT-S trained on TinyImageNet.

Abstract

Data-parallel training relies on frequent gradient synchronization across workers. Standard DDP synchronizes gradients at every iteration, which is effective on fast local-area networks but increasingly sensitive to communication latency and network variability in geographically distributed environments. Periodic methods such as LocalSGD reduce synchronization frequency but rely mainly on parameter averaging, which may be insufficient when worker trajectories diverge. This paper studies synchronization frequency as a systems parameter for communication-constrained distributed training. We evaluate Controlled Periodic Data Parallelism (CPDP), a PyTorch-DDP-compatible strategy that alternates local updates with a reconciliation step combining gradient AllReduce and SlowMo parameter averaging. Experiments are conducted on Grid'5000 across intra-site clusters and a cross-site WAN deployment spanning Nancy and Sophia with 16.6 ms RTT. In the main fixed learning-rate setting of 0.1 on ResNet-50/CIFAR-100, CPDP achieves the highest peak test accuracy among the evaluated configurations. At K=2, CPDP improves over DDP by 2.28 percentage points but incurs additional wall-clock time. At K=4, CPDP improves over DDP by 2.44 percentage points while reducing average wall-clock time by 13.8%. Direct profiling shows that exposed synchronization time at K=4 is roughly half that of DDP, explaining the improved WAN accuracy-time trade-off. Additional experiments on ViT-S/CIFAR-100 and ResNet-50/TinyImageNet show that CPDP remains competitive with DDP and generally improves over LocalSGD. Overall, the results show that synchronization frequency is a practical control parameter for distributed training under communication constraints.

Problem

• data-parallel training
• gradient synchronization
• communication latency
• distributed machine learning
• synchronization frequency

Innovation

• Controlled Periodic Synchronization
• Data-Parallel Training
• Gradient AllReduce
• SlowMo Averaging
• Communication-Constrained Distributed Learning