Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference serving

2026-08-06

AI Summary

This work addresses the challenge that existing LLM inference systems struggle to differentiate the urgency of heterogeneous requests under a uniform SLO, leading to unfair scheduling and suboptimal throughput. To overcome this, the authors propose Cascade, a system that uniquely leverages dynamically estimated latency budgets—computed as the SLO minus the predicted remaining execution time—for both request scheduling and KV cache management across multi-level memory hierarchies. Cascade continuously updates these budgets based on request characteristics, cache states, and system load, guiding scheduling decisions as well as KV cache prefetching, retention, and recomputation policies. Evaluation on three real-world large-model workloads demonstrates that Cascade achieves up to 2.4× higher effective throughput and reduces SLO violations by 40% compared to vLLM’s FCFS scheduler.

Abstract

The reasoning and agentic capabilities of large language models have expanded the range of applications they support, from short interactive exchanges to long, compute-heavy requests. LLM serving platforms today define response-latency service-level objectives, even though requests within the same service can differ by orders of magnitude in input length, generation length, execution cost, and the availability of reusable KV-cache state. As a result, requests governed by the same service level objective have different urgency: after accounting for the time required to execute them, some have substantial latency headroom while others have almost none. We define this headroom---the difference between a request's service level objective and its predicted remaining service time---as its per-request latency budget. We present Cascade, an LLM serving system that estimates and continuously updates this budget from request characteristics, KV-cache state, and current system load. Unlike prior SLO-aware schedulers that use deadlines to govern request ordering alone, Cascade uses a single per-request budget to jointly coordinate request scheduling and KV-cache management across the memory hierarchy. Its scheduler prioritizes requests with little remaining budget, while its memory manager uses the same budget to decide whether non-resident KV state should be restored or prefetched from a deeper tier, retained in HBM, or recomputed. By directing queueing and data-movement overhead toward requests that can absorb it, Cascade improves SLO-satisfied goodput while preserving fairness across heterogeneous request classes. On production traces across three large language models, Cascade improves goodput by up to2.4x and reduces SLO violations by 40% relative to the default vLLM first-come, first-served scheduler.

Problem

• LLM inference serving
• service-level objective
• latency budget
• fairness
• goodput

Innovation

• latency budget
• SLO-aware scheduling
• KV-cache management
• LLM inference serving
• goodput optimization