load balancing

To address the coexistence of performance instability and resource underutilization in multi-application GPU co-location, this paper introduces the first kernel-level, fine-grained resource interference quantification framework spanning multiple hardware layers—including compute units, L1/L2 caches, and memory bandwidth—overcoming the limitations of conventional coarse-grained utilization-based modeling. Leveraging micro-benchmarks, hardware performance counter sampling, kernel-level isolation experiments, and interference modeling, the framework enables reproducible characterization of interference behavior across critical subsystems. Based on this, we design a dynamic co-location scheduler with strict service-level objective (SLO) guarantees, achieving over 35% improvement in aggregate GPU utilization while maintaining quality-of-service requirements. This work establishes both theoretical foundations and empirical validation for predictable, high-performance GPU resource sharing.

GPU clusters suffer from low average utilization (~50%) due to resource fragmentation, heterogeneous workloads, and misalignment between static, single-objective schedulers and dynamic AI demands. Method: We propose a dynamic multi-objective co-scheduling paradigm tailored for multi-tenant AI environments, featuring three integrated schedulers—Hierarchical Priority Scheduler (HPS), Predictive Backfilling Scheduler (PBS), and Smart Batch Scheduler (SBS)—that jointly optimize utilization, fairness, and throughput across mixed training, inference, and research workloads. Our approach innovatively combines dynamic priority assignment, prediction-guided backfilling, intelligent batching, and multidimensional resource modeling, ensuring transparency, interpretability, and production readiness. Results: Experiments demonstrate a GPU utilization increase to 78.2% (+33.2 percentage points), throughput of 25.8 jobs/hour, 92% reduction in task starvation (from 156 to 12 stalled jobs), and fairness variance reduced to 457—significantly outperforming FIFO, SJF, and other baselines.

This work addresses the significant communication bottleneck in multi-GPU training caused by the serial execution of computation and communication. The authors propose a portable runtime mechanism that requires no modifications to vendor libraries or kernels. By dynamically controlling on-chip resource occupancy of compute kernels, elevating the scheduling priority of communication streams, and leveraging shared memory for compute footprint management and cross-GPU resource coordination, the approach effectively enables concurrent execution of computation and collective communication. Evaluated on NVIDIA A40, A100, H100, and AMD MI250X GPUs, the method reduces end-to-end training time by up to 25.5%.

NVIDIA’s Multi-Instance GPU (MIG) technology suffers from severe resource fragmentation, low utilization, and inflexible scheduling in multi-tenant cloud environments due to its static partitioning scheme. Method: This paper proposes the first online, workload-agnostic, fragmentation-aware scheduling framework tailored for MIG. It introduces a lightweight fragmentation metric and employs a greedy strategy to dynamically optimize GPU and MIG slice allocation, minimizing fragmentation growth in real time—without requiring prior knowledge of workload characteristics. Contribution/Results: The framework achieves high scheduling flexibility while maintaining low system overhead. Experiments under high-load conditions show an average 10% improvement in workload admission rate and significant gains in GPU utilization, with total GPU consumption comparable to state-of-the-art baselines. By overcoming MIG’s static partitioning limitation, this work establishes a new paradigm for fine-grained, elastic GPU scheduling in cloud environments.

Distributed deep learning (DL) jobs are highly sensitive to communication latency, yet existing schedulers lack network awareness. To address this, we propose a network-aware GPU cluster scheduler. Our method introduces three key innovations: (1) a job latency-sensitivity–driven GPU resource colocation mechanism, the first of its kind; (2) a network-aware, fine-grained preemption strategy that dynamically prioritizes latency-critical communication; and (3) an adaptive latency timer for automatic, runtime tuning of scheduling parameters. Integrated into a data-driven distributed DL simulation platform, our scheduler achieves up to 69% reduction in end-to-end training time, 83% decrease in average job completion time, and 98% lower communication overhead under high network congestion. Furthermore, it significantly improves GPU resource utilization and overall training efficiency.

This work addresses the unpredictability of GPU kernel execution times caused by inter-kernel data dependencies and resource contention, which undermines real-time guarantees. Focusing on DAG-structured GPU tasks, the paper proposes a scheduling approach that does not rely on kernel priority assumptions. By explicitly modeling kernel dependencies and co-scheduling kernel-level parallelism, the method derives tight and safe worst-case execution time (WCET) bounds. Built upon standard CUDA APIs, the approach requires only DAG task modeling, parallel kernel scheduling, and response time analysis, without additional hardware or software support. Experimental results on both synthetic and real-world benchmarks demonstrate that the proposed method reduces WCET and measured execution time by up to 32.8% and 21.3%, respectively, compared to existing techniques.

This work addresses the inefficiencies in multi-GPU systems caused by suboptimal GPU allocation, which often leads to excessive energy consumption and poor resource utilization. The authors propose EcoSched, an online scheduler that uniquely integrates dynamic GPU count selection with application co-scheduling. EcoSched leverages lightweight runtime performance prediction, a balanced scoring strategy that jointly optimizes energy efficiency and idle resource utilization, and a NUMA-aware task placement mechanism to achieve high scheduling efficacy. Experimental evaluations on H100, A100, and V100 platforms demonstrate that EcoSched reduces energy consumption by up to 14.8%, shortens job completion time by as much as 30.1%, and decreases the energy-delay product (EDP) by up to 40.4%, thereby significantly enhancing both energy efficiency and resource utilization in heterogeneous CPU-GPU systems.

This study addresses the trade-off between energy efficiency and latency for AI/ML workloads in multi-instance GPU (MIG) environments by proposing a dynamic repartitioning scheduling framework tailored to individual MIG instances. The framework integrates scheduler selection with a reinforcement learning–based dynamic repartitioning strategy, marking the first application of reinforcement learning to MIG reconfiguration decisions. It uncovers optimal GPU partitioning patterns under varying temporal and queue-state conditions, enabling predictive and automatic adjustments. Experimental evaluation using real-world diurnal workload traces from data centers demonstrates that the proposed approach improves the combined metric of energy consumption and task latency by 26%, 31%, and 68% compared to twice-daily repartitioning, static partitioning, and no partitioning schemes, respectively.

This study addresses the low utilization of modern GPU computing resources by systematically evaluating the performance, energy efficiency, and resource isolation characteristics of NVIDIA’s Multi-Process Service (MPS) and Multi-Instance GPU (MIG) technologies under concurrent application workloads. The experiments reveal a critical trade-off between MPS’s scheduling flexibility and MIG’s hardware-level isolation: MPS can improve performance by up to 30% and reduce energy consumption by approximately 20% in the absence of memory contention, yet suffers a 30% performance degradation under contention; MIG effectively mitigates resource contention but is constrained by its rigid configuration options and higher overhead. These findings provide empirical foundations for optimizing GPU co-execution strategies driven by application-specific workload characteristics.

This work addresses the low GPU utilization and performance interference caused by coarse-grained resource allocation under mixed HPC, AI, and data analytics workloads. To bridge the granularity gap between static Multi-Instance GPU (MIG) partitions and the actual resource demands of applications, the authors propose a novel co-design mechanism that integrates static MIG partitioning with fine-grained CPU memory offloading, leveraging cache-coherent NVLink-C2C interconnects for the first time. System-level evaluations across representative real-world applications—including NekRS, LAMMPS, Llama3, and Qiskit—demonstrate that the proposed approach significantly reduces GPU idle time, improves throughput and energy efficiency, and effectively mitigates performance degradation due to resource contention in shared environments.