Beyond Greedy Chunking: SLO-Aware Sliding-Window Scheduling for LLM Inference

2026-06-04

AI Summary

This work addresses the challenge of simultaneously achieving high throughput and low user-perceived latency in large language model (LLM) serving, a trade-off exacerbated by existing scheduling methods that impose coarse-grained output constraints, leading to poor resource utilization and inadequate quality-of-service (QoS) guarantees. To overcome these limitations, the authors propose SlidingServe, a novel serving system that introduces a sliding window mechanism to enable SLO-aware dynamic chunked scheduling by leveraging information from both the current and next decoding iterations. SlidingServe further incorporates a multi-level priority scheme to balance fairness and efficiency and employs dynamic programming to construct batches that minimize the risk of SLO violations for critical requests. Experimental results demonstrate that the proposed approach improves service capacity by up to 30% across diverse workloads and reduces SLO violation rates by 16%–53% under heavy load.

Abstract

With the rapid growth of interactive applications in large language model (LLM) online services, maintaining high system throughput while ensuring user-perceived latency has become a key issue in inference scheduling. Existing LLM service systems rely on coarse-grained output constraints, making it difficult to effectively handle resource contention among multiple requests, resulting in low resource utilization efficiency and limited support for fine-grained quality of service (QoS) differentiation. We present SlidingServe, a sliding-window-driven SLO-Aware scheduling system for online LLM inference. SlidingServe designed a lightweight batch latency predictor to estimate the execution time of a batch. Based on this, SlidingServe uses SlidingChunker to combine information from the current iteration and the next iteration to achieve dynamic chunking and improve the overall system throughput while maintaining strict QoS guarantees. SlidingServe introduces Multi-Level Priority Sorter to sort candidate requests in order to balance fairness and efficiency. Additionally, when multiple requests within the same batch are at risk of SLO violating,SlidingServe introduces BatchConstructor, which uses dynamic programming to select the set of requests to execute in the current round, mitigating the SLO violation risk of critical requests.Our evaluation demonstrates that SlidingServe can improve service capacity by up to 30% compared to advanced scheduling systems under various load conditions, and further reduces the rate of SLO violation by 16%-53% under heavy-load inference mode.

Problem

• LLM inference
• SLO-aware scheduling
• resource contention
• quality of service
• online serving

Innovation

• SLO-aware scheduling
• sliding-window inference
• dynamic chunking
• LLM serving
• batch latency prediction