Resource Heterogeneity-Aware and Utilization-Enhanced Scheduling for Deep Learning Clusters

2025-03-13

AI Summary

Existing deep learning training schedulers for heterogeneous AI accelerator clusters (e.g., GPU/TPU) lack fine-grained modeling of hardware-level performance heterogeneity, leading to suboptimal resource utilization and scheduling inefficiency. Method: We propose Hadar, a task-level scheduler, and its enhanced variant HadarE. Their core innovation lies in explicitly incorporating task-level performance heterogeneity into the optimization framework. HadarE further introduces a multi-replica, cross-node concurrent training mechanism to enable coordinated execution across heterogeneous devices while preserving inference accuracy. The approach integrates primal-dual online scheduling, heterogeneous performance feature modeling, and trace-driven evaluation. Results: Experiments show that Hadar achieves 1.20× speedup over the state-of-the-art scheduler Gavel. HadarE reduces total training time by 50% on AWS and 80% on an on-premises cluster, improves resource utilization by 1.45×, and maintains higher model accuracy.

Technology Category

• Machine Learning: Hardware-aware ML
• Planning, Routing, and Scheduling: Learning for Planning and Scheduling
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Scheduling deep learning (DL) models to train on powerful clusters with accelerators like GPUs and TPUs, presently falls short, either lacking fine-grained heterogeneity awareness or leaving resources substantially under-utilized. To fill this gap, we propose a novel design of a task-level heterogeneity-aware scheduler, {em Hadar}, based on an optimization framework that can boost resource utilization. {em Hadar} leverages the performance traits of DL jobs on a heterogeneous DL cluster, characterizes the task-level performance heterogeneity in the optimization problem, and makes scheduling decisions across both spatial and temporal dimensions. %with the objective to reduce the average job completion time of DL jobs. It involves the primal-dual framework employing a dual subroutine, to solve the optimization problem and guide the scheduling design. Our trace-driven simulation with representative DL model training workloads demonstrates that {em Hadar} accelerates the total time duration by 1.20$ imes$ when compared with its state-of-the-art heterogeneity-aware counterpart, Gavel. Further, our {em Hadar} scheduler is enhanced to {em HadarE} by forking each job into multiple copies to let a job train concurrently on heterogeneous GPUs resided on separate available nodes (i.e., machines or servers) for resource utilization enhancement. {em HadarE} is evaluated extensively on physical DL clusters for comparison with {em Hadar} and Gavel. With substantial enhancement in cluster resource utilization (by 1.45$ imes$), {em HadarE} exhibits considerable speed-ups in DL model training, reducing the total time duration by 50% (or 80%) on an Amazon's AWS (or our lab) cluster, while producing trained DL models with consistently better inference quality than those trained by extit{Hadar}.

Problem

• Enhances resource utilization in deep learning clusters.
• Addresses task-level heterogeneity in scheduling DL jobs.
• Reduces job completion time and improves inference quality.

Innovation

• Task-level heterogeneity-aware scheduler design
• Optimization framework boosts resource utilization
• Forking jobs for concurrent training on heterogeneous GPUs