Technology Category
Machine Learning: Distributed Machine Learning & Federated LearningConstraint Satisfaction and Optimization: Distributed CSP/OptimizationSearch and Optimization: Distributed Search
Design Space Exploration of DMA based Finer-Grain Compute Communication Overlap
🤖 AI Summary
In distributed ML, shard-level coupling of communication and computation causes severe performance bottlenecks—up to 1.7× degradation versus ideal throughput—while existing coarse-grained overlap techniques fail to adapt to heterogeneous topologies and fine-grained dataflows. This paper introduces FiCCO (Fine-grained Computation-Communication Overlap), the first systematic framework for operator-level overlap: it decomposes operators within shards, jointly models overlap benefits and operation-level efficiency losses, and employs DMA offloading to mitigate resource contention. FiCCO comprises four core components: fine-grained scheduling modeling, DMA-driven communication offloading, operation-feature-aware heuristic scheduling selection, and efficiency-loss characterization matching. Evaluated on real ML training and inference workloads, FiCCO achieves up to 1.6× speedup over state-of-the-art baselines. Its heuristic scheduler yields optimal schedules in 81% of previously unseen scenarios, demonstrating strong generalization across diverse models, hardware, and network configurations.
Technology Category
Application Category
📝 Abstract
As both ML training and inference are increasingly distributed, parallelization techniques that shard (divide) ML model across GPUs of a distributed system, are often deployed. With such techniques, there is a high prevalence of data-dependent communication and computation operations where communication is exposed, leaving as high as 1.7x ideal performance on the table. Prior works harness the fact that ML model state and inputs are already sharded, and employ careful overlap of individual computation/communication shards. While such coarse-grain overlap is promising, in this work, we instead make a case for finer-grain compute-communication overlap which we term FiCCO, where we argue for finer-granularity, one-level deeper overlap than at shard-level, to unlock compute/communication overlap for a wider set of network topologies, finer-grain dataflow and more. We show that FiCCO opens up a wider design space of execution schedules than possible at shard-level alone. At the same time, decomposition of ML operations into smaller operations (done in both shard-based and finer-grain techniques) causes operation-level inefficiency losses. To balance the two, we first present a detailed characterization of these inefficiency losses, then present a design space of FiCCO schedules, and finally overlay the schedules with concomitant inefficiency signatures. Doing so helps us design heuristics that frameworks and runtimes can harness to select bespoke FiCCO schedules based on the nature of underlying ML operations. Finally, to further minimize contention inefficiencies inherent with operation overlap, we offload communication to GPU DMA engines. We evaluate several scenarios from realistic ML deployments and demonstrate that our proposed bespoke schedules deliver up to 1.6x speedup and our heuristics provide accurate guidance in 81% of unseen scenarios.
Problem
Research questions and friction points this paper is trying to address.
Explores finer-grain compute-communication overlap for distributed ML.
Characterizes inefficiency losses from decomposing ML operations.
Designs heuristics to select optimal schedules using DMA offloading.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Finer-grain compute-communication overlap for wider network topologies
Design space exploration of schedules with inefficiency characterization
Offloading communication to GPU DMA engines to reduce contention
🔎 Similar Papers
No similar papers found.
💼 Related Jobs
