Preserving Admission Responsibility in Multi-Tenant Large Language Model Prefix Caches

2026-08-02

AI Summary

This work addresses the inefficiency in multi-tenant large language model serving where incoming requests frequently evict reusable KV cache blocks belonging to other tenants due to a mismatch between admission control and eviction responsibility. To resolve this, the authors propose PrefixShield, a novel mechanism integrated into vLLM that introduces the concept of persistent caching responsibility. PrefixShield tracks the origin of newly generated KV blocks across requests, restricts reuse when outstanding “debt” remains unfulfilled, and selects eviction candidates based on predicted debt—decoupling value-based ranking from responsibility attribution. Experiments show that with a 4096-block cache, PrefixShield improves victim hit rates by 9.39 and 8.64 percentage points over LRU and S3-FIFO, respectively, recovering hit rates from 4.92% to 84.87%. It maintains a 35.16-percentage-point advantage even under delayed replay scenarios while ensuring fairness, high reuse efficiency, and effective utilization of idle resources.

Abstract

Shared prefix caching turns Graphics Processing Unit (GPU) memory into persistent state shared across Large Language Model (LLM) tenants. A group that materializes new Key-Value (KV) blocks can force another to lose reusable state, yet request-time schedulers account for transient service, replacement policies primarily rank object value, and static partitioning strands idle capacity. We call this mismatch the admission-responsibility gap. To close it, we propose PrefixShield, which meters newly materialized full KV blocks, carries responsibility across requests, gates reuse promotion while debt remains, and uses projected debt to select the group supplying eviction candidates. We implement PrefixShield in vLLM. In paired runs under one-touch pollution, PrefixShield improves victim cache hit ratio by 9.39 percentage points over the Least Recently Used (LRU) policy and 8.64 points over S3-FIFO, restoring the victim from 4.92% to 84.87% at 4096-block scale, and gains 2.00 points over S3-FIFO under two-pass replay. It preserves benign ShareGPT behavior and work-conserving access to idle capacity. Delayed replay yields a 35.16-point advantage while debt remains. These results show that object-value signals rank what to retain, while persistent responsibility determines which group bears reclamation pressure.

Problem

• multi-tenant
• LLM prefix caching
• admission-responsibility gap
• KV cache eviction
• shared GPU memory

Innovation

• prefix caching
• multi-tenant LLM
• admission-responsibility gap
• KV cache management
• PrefixShield