Arachne: Learning to Plan Parallel Training on Dynamic Heterogeneous Clusters

2026-09-28

AI Summary

This study addresses the challenge of explosive search spaces and sluggish responsiveness in parallel planning for heterogeneous GPU clusters, caused by dynamic resources and hardware disparities. To tackle this, we propose a learning-based planner that reduces complex planning to pipeline structure search. Its core innovation is a "template encapsulation" strategy that embeds decisions within structural templates, decoupling core topologies from regularized details. Combined with offline learning, this enables rapid generation of high-throughput plans. The proposed approach significantly compresses the search space and enhances planning efficiency. Experimental results demonstrate throughput improvements of up to 84.5% for dense models and 4.6× for Mixture-of-Experts (MoE) models, comprehensively outperforming five state-of-the-art mainstream planners.

Abstract

Training large machine learning models on shared GPU infrastructures faces two challenges: (1) GPU availability shifts dynamically with varying resource demands from tenants, and (2) hardware heterogeneity accumulates as datacenters continuously adopt new GPU generations. Due to the vast search space induced by heterogeneous GPU types and node sizes, training planners must prune it aggressively to remain tractable, yet must also derive high-throughput plans promptly as cluster configurations change. Arachne achieves this goal through a learning-based planner that reduces the full planning problem to a search over pipeline structures. Arachne encapsulates planning decisions in a structural template and learns to construct plans from templates over diverse cluster configurations offline. This design is effective because structural decisions constitute the performance-critical core of a parallelism plan, while the rest follows by rule or from a small priced candidate set once the plan structure is fixed. Evaluation shows that Arachne matches or exceeds the best plan found by five existing planners across clusters with varying GPU types and node sizes for three models of different sizes by up to 84.5% in throughput on dense models and 4.6x on MoE models.

Problem

• Heterogeneous GPU clusters
• Parallelism planning
• Large model training
• Dynamic resource availability
• Search space pruning

Innovation

• Parallelism Planning
• Heterogeneous GPU Clusters
• Structural Templates
• Learning-based Planner
• Pipeline Structures