cache management

Existing KV cache eviction strategies for LLM inference services—particularly generic policies like LRU—suffer from poor adaptability to workload characteristics, leading to suboptimal performance. Method: This paper presents the first empirical analysis of KV caching behavior using real-world cloud service traces, revealing strong skewness, intra-class predictability, and low capacity sensitivity in KV reuse across single- and multi-turn requests. Based on these insights, we propose a workload-aware dynamic eviction policy that jointly models token热度 (access frequency) and时效 (temporal recency), validated via analytical cache modeling, offline trace replay, and online A/B testing. Contribution/Results: On production traces, our approach achieves up to a 23% higher cache hit rate than LRU; under memory-constrained conditions, it reduces end-to-end latency by 18% and increases throughput by over 15%, significantly enhancing deployment efficiency in practical LLM serving systems.

This work addresses the inherent tension between cache hit rate and load balancing in memory-constrained multi-LLM serving systems, where conventional LRU-based KV cache eviction struggles under dynamic workloads. The paper presents the first theoretical framework for jointly optimizing KV cache eviction and query routing, introducing a协同 algorithm that integrates theoretically grounded randomized KV cache eviction with online learning–driven adaptive query routing. The approach further supports prefix-sharing across multiple models to enhance cache efficiency. Extensive experiments across four benchmarks and three prefix-sharing configurations demonstrate substantial improvements: up to 6.92× higher cache hit rate, 11.96× lower latency, 14.06× reduction in time-to-first-token, and a 77.4% increase in throughput.

This work addresses the inefficiency of existing large language model (LLM) serving systems in handling coding agent workloads, which exhibit frequent prefix repetitions and sustained pressure on key-value (KV) cache capacity. By analyzing real-world coding assistant interaction traces, the authors propose CacheWise, a novel system that integrates prefix-aware scheduling with a lightweight, tool-call metadata–driven predictive cache eviction policy. This approach substantially enhances KV cache reuse without compromising response quality. Implemented atop vLLM, CacheWise demonstrates significant performance gains under realistic workloads, reducing KV cache evictions by 2–2.6× and accelerating overall session completion time by up to 3.5× compared to baseline systems.

To address high KV cache generation overhead during the prefill phase and I/O-bandwidth-limited prefix caching—both contributing to elevated time-to-first-token (TTFT) in long-context LLM inference—this paper proposes a novel parallel, cooperative KV cache scheduling paradigm. Our method introduces: (1) a bidirectional dynamic scheduling strategy that balances compute and I/O resources in real time; (2) an adaptive scheduling mechanism unifying support for both prefix and non-prefix requests to jointly optimize throughput and latency; and (3) a system-level KV cache framework integrating resource awareness, compute-I/O overlap, and joint request scheduling. Evaluated across diverse hardware platforms, datasets, and heterogeneous storage configurations, our approach achieves a 2.6× average reduction in TTFT, significantly enhancing inference efficiency for long-context online LLM services.

Existing uniform layer-wise KV cache eviction schemes fail to accommodate inter-layer variations in attention patterns and temporal importance decay, leading to memory explosion in long-context reasoning for large language models. Method: We propose a hierarchical preference-driven cascaded adaptive cache eviction mechanism: (1) a novel layer-preference-aware global cache allocation paradigm modeled as dynamic “cake-cutting”-style resource partitioning; (2) a dynamic weighted eviction scoring metric integrating spatiotemporal attention analysis; and (3) a cascaded memory-constrained management framework enabling deep synergy with FlashAttention-2. Results: On LongBench and NeedleBench, our method retains full model performance using only 3.2% of the original KV cache, reduces decoding latency by over 10× for 128K-context inputs, and significantly outperforms state-of-the-art methods under stringent memory constraints.

This work addresses the lack of a portable contract mechanism in existing systems for governing the reuse of resident key-value (KV) cache entries under memory pressure, which leads to ineffective arbitration when conflicts arise with active KV requests. The paper proposes a declarative mechanism for resident KV entries that binds reuse intent, predicates, lifecycle state, and feasibility outcomes into a runtime consistency contract, enabling traceable arbitration between active and resident KV entries. It introduces, for the first time, a telemetry-aware, declaration-level contract that transforms invisible resident KV evictions into reconstructible arbitration events, supporting scheduler-visible rejections with clear attribution. Prototype experiments based on vLLM demonstrate that the approach successfully handles 60 resident declarations alongside 70 active requests within an 80-block KV pool, achieving precise attribution and proactive rejection—outperforming conventional cache replacement policies.

This work addresses the high memory cost of KV caching in large language model inference, where existing eviction strategies often rely on heuristics lacking theoretical grounding. The authors introduce, for the first time, the information bottleneck principle into KV cache management by constructing a linear Gaussian attention proxy model. This framework yields a mutual information–based objective function quantifying information capacity, leading to CapKV—an information-aware eviction method that directly optimizes the amount of useful information retained in the cache. The analysis reveals that several existing strategies are implicit approximations of this principle. Experiments across multiple models and long-context benchmarks demonstrate that CapKV consistently outperforms prior methods, achieving a superior trade-off between memory efficiency and generation fidelity.

This work addresses the inefficiency of conventional LRU-based KV cache management under diverse inference workloads, where inter-task eviction degrades cache performance. To mitigate this, the study introduces a frequency-aware mechanism into KV cache management and proposes an adaptive algorithm that dynamically allocates cache space by jointly considering recency and frequency of KV blocks. Integrated into the vLLM inference framework, the method significantly enhances cache hit rates and inference efficiency: on synthetic document QA tasks, it achieves up to a 10.8% improvement in cache hit rate and a 12.6% reduction in time-to-first-token latency; on real-world dialogue tasks, it yields gains of 2.1% and 2.0%, respectively. The approach also demonstrates strong interpretability and scales effectively with batched inference.

Existing KV-cache management approaches struggle to handle context-dependent invocation sequences in dynamic agent workflows, resulting in inefficient cache reuse. This work proposes PBKV, the first system to introduce predictive mechanisms into dynamic agent scenarios. PBKV fuses historical workflow patterns with the current task context to forecast multi-step future agent calls and leverages these predictions to evaluate the reuse value of cached entries, enabling context-aware intelligent cache retention. To enhance robustness, the system conservatively utilizes prediction outcomes during both cache eviction and prefetching. Experimental results demonstrate that PBKV significantly outperforms existing methods across three workflow benchmarks: it achieves up to 1.85× speedup over LRU in dynamic workflows and up to 1.26× speedup over the state-of-the-art KVFlow in static workflows.

Existing KV cache mechanisms only support append-only operations or kernel-managed updates, failing to accommodate the need for agent-driven, policy-guided active editing of cached context—leading to inefficient recomputation whenever prefix modifications are required. This work proposes Leyline, a server-side primitive that decouples editing intent from positional correctness maintenance via a declarative four-tuple instruction format, enabling the first externally controllable KV cache editing framework. Leyline’s architecture-agnostic interface, combined with closed-form RoPE rotation correction, supports both in-place concatenation and prefix truncation with refilling, and integrates seamlessly into diverse hardware kernels. Experiments demonstrate that Leyline improves cache hit rates by 11.2 percentage points and reduces latency by up to 241 ms; on debug-gym tasks, it boosts solution success rates by 14.3 percentage points using only ten lines of truncation rules.