Agentic CPU-GPU Scheduling for Heterogeneous AI Workloads

2026-07-24

AI Summary

This work addresses the inefficiency of existing AI systems that indiscriminately schedule tasks to GPUs in heterogeneous CPU/GPU environments, overlooking scenarios where CPU execution is superior or device-neutral, thereby causing resource waste and increased latency. To overcome this, we propose a dynamic three-way scheduling mechanism—choosing among immediate GPU execution, queued GPU execution, or CPU offloading—that integrates an LLM-based agent with a runtime monitor to optimize scheduling decisions in real time under VRAM constraints, without requiring offline training or exhaustive search. By dynamically sensing GPU utilization and VRAM contention, our approach transcends the limitations of static performance profiling. Experiments demonstrate that it achieves optimal solutions matching brute-force search across 13 diverse scenarios—including serial, parallel, and VRAM-constrained workloads—while attaining scheduling accuracy on par with the best classical baselines and significantly outperforming HEFT, StarPU, and all-GPU strategies.

Abstract

Agentic AI systems compose heterogeneous tool workloads on shared GPU/CPU infrastructure, yet existing frameworks assign all GPU-capable tools to the GPU by default. We profile 19 AI tools across GPU and CPU and find that 11 are GPU-preferred, 4 are ambiguous, 1 is CPU-preferred due to PCIe transfer dominance, and 3 are device-neutral, establishing that blanket GPU-first scheduling is suboptimal. We formulate device scheduling as assigning each tool to one of three options: immediate GPU execution, queued GPU execution, or CPU offload, under a VRAM budget, and identify two runtime factors that cause end-to-end latency to diverge from static profiles: GPU utilization contention and VRAM capacity contention. We present an agentic scheduler that pairs an LLM agent with an algorithmic runtime monitor, where the monitor expands what the LLM can observe via running averages, symmetric reprobing, swap reprobing, and exploration hints, without ever prescribing which mapping to adopt. Across 13 scenarios spanning serial execution, parallel contention, and memory-constrained execution, the agentic scheduler reaches the brute-force optimal mapping in all 13 scenarios, matching the best classical baseline on mapping accuracy while avoiding bandit-style exploration over complete mappings, and outperforming HEFT, StarPU, and the all-GPU policy while requiring zero offline training.

Problem

• Agentic AI
• Heterogeneous scheduling
• GPU-CPU allocation
• VRAM contention
• End-to-end latency

Innovation

• Agentic Scheduling
• Heterogeneous AI Workloads
• CPU-GPU Orchestration
• Runtime-Aware Scheduling
• LLM-Augmented Systems