Fairness-Aware and Latency-Controllable Scheduling for Chunked-Prefill LLM Serving

2026-06-08

AI Summary

This work addresses the inefficiencies of existing large language model serving systems under highly heterogeneous workloads, where first-come-first-served (FCFS) scheduling and static token budgets often lead to unfairness, high latency variability, head-of-line blocking, and request starvation. To overcome these limitations, the authors propose a novel scheduling framework that jointly optimizes fairness and latency predictability by integrating a time-aware aging mechanism, latency-predictive request scheduling (LPRS), and adaptive prefill control (APC). Instead of relying on fixed token budgets, APC dynamically adjusts prefill concurrency based on target latency constraints. Experimental results demonstrate that, compared to FCFS, the proposed approach reduces average end-to-end latency by over 10% and substantially lowers P99 tail latency and prefill fragmentation across both NVIDIA GPU and Ascend accelerator platforms, showcasing its effectiveness and hardware generality.

Abstract

As large language models (LLMs) are increasingly deployed with highly heterogeneous workloads, chunked-prefill execution has emerged as a mainstream serving architecture. Balancing scheduling fairness and latency stability in such environments is critical; otherwise, severe head-of-line blocking and request starvation will degrade user experience. However, existing systems rely on rigid First-Come, First-Served (FCFS) policies and static token budgets, leading to fairness degradation and unpredictable latency jitter. To address these issues, we propose a fairness-aware and latency-controllable scheduling framework for chunked-prefill LLM engines. Specifically, we design a lightweight aging-based scheduling policy that dynamically calculates priorities using accumulated waiting time and remaining prefill work. Furthermore, we develop Latency-Prediction-Based Request Scheduling (LPRS) and Active Prefill Control (APC) to replace static budgets with target-time constraints and actively regulate prefill concurrency. We evaluated our scheduling framework on NVIDIA GPUs and Ascend accelerators using real-world workloads. Results show the aging policy reduces mean end-to-end latency by over 10\% compared to FCFS. Moreover, LPRS and APC significantly reduce P99 tail latency and suppress prefill fragmentation, confirming that the structural prefill control and the temporal latency constraints are fundamentally complementary. All codes have been released in Github.

Problem

• fairness
• latency stability
• chunked-prefill
• LLM serving
• request starvation

Innovation

• chunked-prefill
• fairness-aware scheduling
• latency-controllable
• aging-based scheduling
• prefill concurrency control