DBLP: Phase-Aware Bounded-Loss Transport for Burst-Resilient Distributed ML Training

2026-05-03

AI Summary

This work addresses the challenge of tail latency and training time variability in distributed machine learning caused by transient network congestion, a problem exacerbated by existing communication protocols' lack of awareness of training semantics. To this end, the paper proposes the Dynamic Bounded Loss Protocol (DBLP), which uniquely integrates training-phase awareness with a gradient loss-tolerance mechanism. DBLP dynamically adapts gradient compression and dropping strategies across different training stages, achieving robustness to microbursts while preserving model accuracy. Experimental results demonstrate that DBLP reduces end-to-end training time by 24.4% on average (up to 33.9%) compared to state-of-the-art approaches, accelerates per-iteration communication latency by up to 5.88× under microburst conditions, and maintains comparable test accuracy.

Abstract

Distributed machine learning (ML) training has become a necessity with the prevalence of billion to trillion-parameter-scale models. While prior work has improved training efficiency from the ML perspective at the application layer, it often fails to address transient congestion events at the network layer that introduce severe tail latency and training-time variability, thereby undermining the quality of service (QoS) of distributed ML training systems. Existing network optimizations treat all gradients equally and thus fail to integrate sufficient model-training insights into communication protocol design. In this paper, we present Dynamic Bounded-Loss Protocol (DBLP), a burst-resilient, training-phase-aware, and hardware-agnostic transport protocol that incorporates model-level tolerance properties into gradient communication. By dynamically adjusting gradient loss tolerance across training phases, DBLP reduces overall training time and mitigates tail-latency collapse during transient high-loss events (i.e., microbursts). Compared to the current state-of-the-art solution (baseline), DBLP tolerates significantly higher loss while achieving comparable test accuracy, and reduces end-to-end training time by an average of 24.4% and a maximum of 33.9%. At microburst events, DBLP achieves up to 5.88x single-round communication latency speedups over the baseline, preventing burst-induced tail-latency spikes and maintaining stable training performance.

Problem

• distributed machine learning
• transient congestion
• tail latency
• gradient communication
• burst resilience

Innovation

• burst-resilient
• phase-aware
• bounded-loss transport
• distributed ML training
• gradient communication