Requests of a Feather Must Flock Together: Batch Size vs. Prefix Homogeneity in LLM Inference

2026-05-07

AI Summary

This work addresses the inefficiency of traditional batching strategies in large language model inference, which neglect request prefix homogeneity and consequently suffer from poor KV cache access patterns and limited throughput. The authors propose the first reinforcement learning–based, prefix-aware scheduler that dynamically balances batch size against prefix sharing. To enable efficient prefix matching and request selection, they introduce a novel Chunked Hash Tree data structure, significantly enhancing both spatial and temporal locality of KV cache accesses. Integrated into vLLM and SGLang inference engines, the method achieves 2–10× higher end-to-end throughput over existing schedulers on workloads with shared prefixes, while maintaining baseline-level performance in scenarios without prefix sharing.

Abstract

Auto-regressive token generation in large language models is memory-bound because it requires "attending to" key and value tensors (KV cache) of all previous tokens. Prior work aims to improve the efficiency of this decode process by batching multiple requests together, and maximizing batch size subject to GPU memory constraints. The key observation of our work is that with prefix-sharing workloads, smaller, prefix-homogeneous batches -- where all requests share a common prefix -- can achieve higher decode throughput than larger, heterogeneous batches, due to better spatial and temporal locality during KV cache accesses. However, prefix-aware schedulers in state-of-the-art inference engines maximize prefix reuse within a batch only to reduce KV cache memory footprint, but do not stop batch formation at smaller homogeneous batches that could have performed better. Further, we show that shared prefix detection in existing schedulers relies on radix-tree traversals, incurring substantial CPU overhead that is often comparable to GPU execution time. This paper presents Feather, a prefix-aware scheduler that uses reinforcement learning (RL) to learn the optimal tradeoff between batch size and prefix homogeneity. We also introduce Chunked Hash Tree (CHT), a lightweight data structure that enables fast prefix detection and efficient request selection for the RL scheduler, avoiding expensive tree traversals. We integrate Feather into vLLM and SGLang, and our evaluation shows that Feather achieves 2--10$\times$ higher end-to-end throughput as compared to existing schedulers, while doing no worse than the status quo when the workload does not have enough prefix sharing. Feather achieves these gains by reducing the total number of KV cache accesses, surpassing the performance of prefix-aware attention kernels that have the same goal.

Problem

• LLM inference
• batch scheduling
• prefix sharing
• KV cache
• throughput optimization

Innovation

• prefix homogeneity
• KV cache locality
• reinforcement learning scheduler
• Chunked Hash Tree
• LLM inference batching