Maestro: Workload-Aware Cross-Cluster Scheduling for LLM-Based Multi-Agent Systems

2026-06-11

AI Summary

This work addresses the challenges of deploying large language model (LLM) multi-agent systems at scale, including high resource consumption, memory fragmentation, cold-start latency, and inefficient cross-cluster scheduling. The paper presents the first scheduling system that integrates agent semantics and role information to predict output lengths and memory requirements across workflow stages, thereby enabling a hierarchical scheduling mechanism. Key innovations include workflow-aware priority scheduling, elastic memory allocation, latency-aware cross-cluster routing, and workflow-level queue management. Experimental results demonstrate that the proposed system reduces HBM memory usage by 67.2% in both prototype deployment and trace-driven simulation, and improves SLO compliance by 23.6 percentage points over EDF under highly contended scenarios.

Abstract

Large Language Model based Multi-Agent Systems (LLM-MAS) have emerged as a powerful paradigm for tackling complex tasks by breaking them into collaborative workflows of specialized LLM-powered agents. However, deploying such multi-agent workloads at scale poses significant system challenges. Each user query spawns an iterative pipeline of LLM calls, greatly amplifying resource consumption compared to single-turn queries. In resource-constrained cloud settings, these workflows face non-deterministic and input-dependent costs at decode stage, heavy-tailed multi-model requirements with memory fragmentation and over-provisioning, and cross-cluster scheduling trade-offs. We present Maestro, a workload-aware scheduling system designed for LLM-MAS serving under strict GPU budgets. Maestro explicitly leverages agent semantics and roles: it predicts the output length and memory usage of each stage and uses this prediction to drive a hierarchical scheduler. At the node level, Maestro enables dynamic multi-model co-location via hierarchical weight caching and elastic memory provisioning. At the cluster level, it performs latency-aware routing to avoid cold-start delays and memory overloads. At the global level, it enforces workflow-aware prioritization to minimize head-of-line blocking for interactive tasks. Across prototype experiments and trace-driven simulations, Maestro reduces KV-reservation HBM by 67.2% and improves high-contention SLO attainment over EDF by 23.6 percentage points.

Problem

• LLM-MAS
• resource-constrained scheduling
• memory fragmentation
• cross-cluster scheduling
• workload variability

Innovation

• workload-aware scheduling
• LLM-based multi-agent systems
• hierarchical weight caching
• elastic memory provisioning
• cross-cluster orchestration