Taming Request Imbalance: SLO-Aware Scheduling for Disaggregated LLM Inference

2026-05-04

AI Summary

This work addresses the challenge of meeting stringent service-level objectives (SLOs) in disaggregated large language model inference, where the long-tailed distribution of request lengths causes head-of-line blocking during prefill and low resource utilization during decode. To this end, the paper introduces Kairos, the first SLO-aware scheduler for disaggregated inference. Kairos employs deadline-predictive urgency-based scheduling on the prefill side and a slack-time-guided adaptive continuous batching strategy on the decode side, jointly optimizing SLO compliance for time-to-first-token (TTFT) and time-per-output-token (TPOT). Experiments show that, compared to the best existing baseline, Kairos improves TTFT, TPOT, and end-to-end SLO attainment by up to 23.9%, 27.1%, and 33.8%, respectively, while increasing decode throughput by 19.3%.

Abstract

In production environments, large language model (LLM) serving is required to meet stringent service-level objectives (SLOs) amid highly variable request patterns. In practice, request lengths follow a long-tail distribution, which gives rise to head-of-line blocking on the prefill side and underutilization caused by stragglers on the decode side in disaggregated serving architectures. Current systems, which adopt first-come-first-served (FCFS) scheduling for prefill and continuous batching for decode, lack the ability to adapt to this imbalance, resulting in compromised SLO attainment and reduced throughput. To address these challenges, we propose Kairos, an SLO-aware scheduling system equipped with two complementary mechanisms. On the prefill side, Kairos employs urgency-based priority scheduling: it predicts prefill completion times and dynamically selects requests to maximize the attainment of time-to-first-token (TTFT) SLOs. On the decode side, Kairos introduces slack-guided adaptive batching, which leverages the gap between per-step decode time and the time-per-output-token (TPOT) SLO to greedily pack short requests. This approach maximizes throughput while strictly adhering to SLO requirements. We implement Kairos and conduct evaluations using an online serving dataset and a state-of-the-art LLM. Experimental results demonstrate that, compared with state-of-the-art baselines, Kairos improves TTFT SLO attainment by up to 23.9\%, TPOT SLO attainment by up to 27.1\%, end-to-end SLO attainment by up to 33.8\%, and decode throughput by up to 19.3\%.

Problem

• request imbalance
• SLO-aware scheduling
• disaggregated LLM inference
• head-of-line blocking
• straggler underutilization

Innovation

• SLO-aware scheduling
• disaggregated LLM inference
• urgency-based priority scheduling
• slack-guided adaptive batching
• request imbalance