Pipelined Gradient Coding

2026-07-22

AI Summary

This work addresses the training latency caused by stragglers in large-scale distributed machine learning by proposing a novel pipelined gradient coding scheme. It introduces computation pipelining into the gradient coding framework for the first time, enabling each worker node to process only one data partition per iteration step. By integrating two efficient data placement strategies—fractional repetition (FR) and cyclic repetition (CR)—the method reduces per-step computational load while ensuring fault-tolerant training. Theoretical analysis provides convergence guarantees, and experimental results demonstrate that the proposed approach significantly reduces training time and accelerates convergence compared to existing gradient coding and baseline methods, both in simulated environments and real-world cloud platforms.

Abstract

In large-scale machine learning, distributed training commonly involves multiple workers evaluating the gradients of the model on different dataset partitions. A common challenge is the presence of straggling workers, which may significantly slow down training. Traditional gradient coding (GC) addresses this by duplicating dataset partitions across workers, allowing for the replacement of missing gradients from stragglers. However, GC requires workers to evaluate gradients on multiple dataset partitions in each step, potentially increasing overall training time. In this paper, we propose to pipeline GC, such that gradient evaluation is segmented across multiple steps and each worker evaluates gradients on just a single dataset partition per step. We develop the pipelined version for fractional repetition (FR) and cyclic repetition (CR), two representative dataset placement schemes in GC, and prove convergence guarantees for both. Through extensive simulations and experiments on cloud infrastructure, our schemes not only significantly reduce training time but also accelerate convergence compared to GC and other baselines.

Problem

• straggler mitigation
• distributed training
• gradient coding
• training efficiency
• large-scale machine learning

Innovation

• Pipelined Gradient Coding
• Straggler Mitigation
• Distributed Training
• Fractional Repetition
• Cyclic Repetition