Unifying Local Communications and Local Updates for LLM Pretraining

2026-06-09

AI Summary

This work addresses the communication bottleneck of All-Reduce in large model pretraining under cross-cluster, low-bandwidth, and heterogeneous compute environments. The authors propose GASLoC, the first decentralized pretraining framework supporting adaptive optimizers, which unifies communication and computation through a novel outer-loop optimizer design, a sparse random gossip communication topology, and a multi-step local update mechanism. GASLoC achieves superior performance over existing decentralized methods with a single communication step on standard LLM tasks and significantly outperforms DiLoCo under multi-local-step and heterogeneous bandwidth scenarios, enabling efficient and scalable training.

Abstract

Communication-efficient pre-training of LLMs is increasingly important as training draws on compute distributed across clusters, data centers, and lower-bandwidth links. Many practical methods reduce communication frequency but still rely on synchronous All-Reduce operations that maintain identical model states and tie progress to global collectives. This can become a bottleneck when bandwidth or worker speed is heterogeneous. We introduce GASLoC, a novel decentralized pre-training algorithm that generalizes the notion of communication acceleration to the recently popular "outer optimizer" to allow a practical gossip-based training framework that is compatible with adaptive optimizers, allows for local optimizer steps, and can utilize sparse randomized peer communication. Empirically, on a number of standard LLM training tasks, we demonstrate that GASLoC outperforms state-of-the-art decentralized algorithms in single step per communication setting for a number of topologies and, unlike existing decentralized methods in the LLM setting, it allows to obtain performance competitive with DiLoCo when utilizing multiple local steps. In the heterogeneous bandwidth setting we demonstrate the advantage of GASLoC showing that it can significantly outperform DiLoCo.

Problem

• LLM pretraining
• communication efficiency
• decentralized training
• heterogeneous bandwidth
• synchronous All-Reduce

Innovation

• decentralized training
• gossip-based communication
• local updates
• communication efficiency
• outer optimizer