🤖 AI Summary
NVIDIA’s Multi-Instance GPU (MIG) technology suffers from severe resource fragmentation, low utilization, and inflexible scheduling in multi-tenant cloud environments due to its static partitioning scheme.
Method: This paper proposes the first online, workload-agnostic, fragmentation-aware scheduling framework tailored for MIG. It introduces a lightweight fragmentation metric and employs a greedy strategy to dynamically optimize GPU and MIG slice allocation, minimizing fragmentation growth in real time—without requiring prior knowledge of workload characteristics.
Contribution/Results: The framework achieves high scheduling flexibility while maintaining low system overhead. Experiments under high-load conditions show an average 10% improvement in workload admission rate and significant gains in GPU utilization, with total GPU consumption comparable to state-of-the-art baselines. By overcoming MIG’s static partitioning limitation, this work establishes a new paradigm for fine-grained, elastic GPU scheduling in cloud environments.
📝 Abstract
The explosive growth of AI applications has created unprecedented demand for GPU resources. Cloud providers meet this demand through GPU-as-a-Service platforms that offer rentable GPU resources for running AI workloads. In this context, the sharing of GPU resources between different tenants is essential to maximize the number of scheduled workloads. Among the various GPU sharing technologies, NVIDIA's Multi-Instance GPU (MIG) stands out by partitioning GPUs at hardware level into isolated slices with dedicated compute and memory, ensuring strong tenant isolation, preventing resource contention, and enhancing security. Despite these advantages, MIG's fixed partitioning introduces scheduling rigidity, leading to severe GPU fragmentation in multi-tenant environments, where workloads are continuously deployed and terminated. Fragmentation leaves GPUs underutilized, limiting the number of workloads that can be accommodated. To overcome this challenge, we propose a novel scheduling framework for MIG-based clouds that maximizes workload acceptance while mitigating fragmentation in an online, workload-agnostic setting. We introduce a fragmentation metric to quantify resource inefficiency and guide allocation decisions. Building on this metric, our greedy scheduling algorithm selects GPUs and MIG slices that minimize fragmentation growth for each incoming workload. We evaluate our approach against multiple baseline strategies under diverse workload distributions. Results demonstrate that our method consistently achieves higher workload acceptance rates, leading to an average 10% increase in the number of scheduled workloads in heavy load conditions, while using approximately the same number of GPUs as the benchmark methods.