PrefixPlace: Provable Prefix Key-Value Placement for Large Language Model Serving under Heterogeneous Compute and Transfer Costs

2026-08-02

AI Summary

This work addresses the inefficiency in large language model inference caused by suboptimal placement of prefix key-value (KV) caches, where cache misses necessitate recomputation or remote retrieval. Existing strategies overlook the heterogeneous costs of computation and data transfer across diverse hardware, leading to degraded performance. To remedy this, the paper introduces PrefixPlace, the first framework that explicitly incorporates heterogeneous cost models into prefix KV placement decisions. The problem is formulated as a constrained monotone facility location problem, for which the authors design an approximation algorithm based on tree-structured dynamic programming with O(nk) time complexity, enhanced by multi-start coordinate descent to improve solution quality while preserving theoretical guarantees. Experiments demonstrate that PrefixPlace achieves 99.84% of the optimal cost on average across 432 instances with known optima, reduces total cost by 40.3% and 40.4% over vLLM-APC on RAG replay and WikiQA benchmarks respectively, and scales to plan placements for 50,000 nodes across 16 workers within 12.3 seconds.

Abstract

Prefix Key-Value (KV) reuse avoids repeated prefill in Large Language Model (LLM) inference, but local misses require recomputation or replica fetches. Their relative cost varies with hardware, prefix depth, KV goodput, and replica location, making hit-rate-based placement suboptimal. To address this issue, we propose an epoch-level planner, PrefixPlace, which assigns prefix-complete targets under memory budgets and profiled demand, compute, and transfer costs. The objective decomposes into local-copy value plus first-replica coverage, and source-dependent costs yield a monotone facility-location objective; each worker update is an additive rooted-tree problem solved exactly in O(nk) time for n chunks and capacity k, giving a fixed-order 1/2-approximation that coordinate refinement and order-diverse starts improve without weakening. T4, L4, and A100 measurements reveal distinct regimes. Across 432 instances with exact optima, PrefixPlace averages 99.84% of optimum and never falls below 98.02%. In Retrieval-Augmented Generation (RAG) replays, it improves materialization-cost saving by 40.3% over vLLM Automatic Prefix Caching (vLLM-APC) and 6.3% over the best offline baseline. On WikiQA, gains are 40.4% and 5.3%. Finally, PrefixPlace solves a 50,000-node, 16-worker placement in 12.3 s on one processor, enabling timely replanning.

Problem

• Prefix KV placement
• LLM serving
• heterogeneous costs
• cache miss
• memory budget

Innovation

• Prefix KV placement
• heterogeneous compute
• facility location
• LLM serving
• cost-aware caching