🤖 AI Summary
GPU clusters suffer from low average utilization (~50%) due to resource fragmentation, heterogeneous workloads, and misalignment between static, single-objective schedulers and dynamic AI demands. Method: We propose a dynamic multi-objective co-scheduling paradigm tailored for multi-tenant AI environments, featuring three integrated schedulers—Hierarchical Priority Scheduler (HPS), Predictive Backfilling Scheduler (PBS), and Smart Batch Scheduler (SBS)—that jointly optimize utilization, fairness, and throughput across mixed training, inference, and research workloads. Our approach innovatively combines dynamic priority assignment, prediction-guided backfilling, intelligent batching, and multidimensional resource modeling, ensuring transparency, interpretability, and production readiness. Results: Experiments demonstrate a GPU utilization increase to 78.2% (+33.2 percentage points), throughput of 25.8 jobs/hour, 92% reduction in task starvation (from 156 to 12 stalled jobs), and fairness variance reduced to 457—significantly outperforming FIFO, SJF, and other baselines.
📝 Abstract
GPU clusters have become essential for training and deploying modern AI systems, yet real deployments continue to report average utilization near 50%. This inefficiency is largely caused by fragmentation, heterogeneous workloads, and the limitations of static scheduling policies. This work presents a systematic evaluation of these issues and introduces three specialized dynamic schedulers: Hybrid Priority (HPS), Predictive Backfill (PBS), and Smart Batch (SBS). These schedulers are designed to improve utilization, fairness, and overall throughput in multi-tenant GPU clusters. We evaluate all schedulers using a controlled simulation of 1,000 AI jobs on a 64-GPU, 8-node cluster that includes a realistic mix of training, inference, and research workloads. Static baselines (FIFO, SJF, Shortest, Shortest-GPU) achieve 45 to 67% GPU utilization and 12.5 to 18.3 jobs per hour and experience severe starvation, with as many as 156 jobs waiting longer than 30 minutes. The dynamic schedulers significantly outperform these policies. HPS achieves the highest utilization (78.2%), highest throughput (25.8 jobs per hour), and the lowest fairness variance among dynamic methods (457), reducing starvation to 12 jobs. PBS improves fragmentation handling and reaches 76.1% utilization, while SBS increases efficiency for structurally similar jobs and reaches 74.6% utilization. Across all key metrics, including throughput, job wait times, fairness variance, and starvation, dynamic multi-objective schedulers consistently outperform single-objective heuristics. These results show that targeted and transparent scheduling strategies can meaningfully increase GPU efficiency in heterogeneous AI clusters and provide a practical foundation for future production scheduling frameworks.