eviction policy design

Existing KV cache eviction strategies for LLM inference services—particularly generic policies like LRU—suffer from poor adaptability to workload characteristics, leading to suboptimal performance. Method: This paper presents the first empirical analysis of KV caching behavior using real-world cloud service traces, revealing strong skewness, intra-class predictability, and low capacity sensitivity in KV reuse across single- and multi-turn requests. Based on these insights, we propose a workload-aware dynamic eviction policy that jointly models token热度 (access frequency) and时效 (temporal recency), validated via analytical cache modeling, offline trace replay, and online A/B testing. Contribution/Results: On production traces, our approach achieves up to a 23% higher cache hit rate than LRU; under memory-constrained conditions, it reduces end-to-end latency by 18% and increases throughput by over 15%, significantly enhancing deployment efficiency in practical LLM serving systems.

This work addresses the limitations of traditional Linux page cache eviction policies, which rely on fixed heuristics and struggle to adapt to diverse workloads, thereby constraining cache efficiency. The authors propose the first integration of a lightweight single-layer perceptron directly into the kernel’s page cache subsystem, leveraging eBPF to enable low-overhead, real-time intelligent eviction decisions. The model is trained on kernel-level data collected from real-world workloads to predict page reuse times and dynamically select eviction candidates. Experimental results across a range of representative workloads demonstrate that, compared to a FIFO baseline, the approach improves cache hit rates by up to 10% and achieves a median AUC of 80%, confirming the feasibility and superiority of machine learning–driven cache management within the kernel.

Public cloud virtual machines (VMs) lack low-level CPU cache visibility and control, rendering conventional cache optimization techniques ineffective. To address this, we propose CacheX—the first hardware- and hypervisor-agnostic, purely software-based fine-grained cache abstraction and probing framework for VMs. CacheX leverages eviction sets to accurately infer last-level cache (LLC) topology and contention states. It further introduces an LLC contention-aware task scheduler and a virtual-coloring–aware page cache management mechanism. The entire solution is fully integrated into the x86 Linux kernel. Evaluation across mainstream public cloud VMs demonstrates that CacheX improves average cache utilization by 32.7% and boosts performance of critical workloads by 14.2%–28.5%, effectively overcoming a fundamental bottleneck in cloud environment cache optimization.

Existing uniform layer-wise KV cache eviction schemes fail to accommodate inter-layer variations in attention patterns and temporal importance decay, leading to memory explosion in long-context reasoning for large language models. Method: We propose a hierarchical preference-driven cascaded adaptive cache eviction mechanism: (1) a novel layer-preference-aware global cache allocation paradigm modeled as dynamic “cake-cutting”-style resource partitioning; (2) a dynamic weighted eviction scoring metric integrating spatiotemporal attention analysis; and (3) a cascaded memory-constrained management framework enabling deep synergy with FlashAttention-2. Results: On LongBench and NeedleBench, our method retains full model performance using only 3.2% of the original KV cache, reduces decoding latency by over 10× for 128K-context inputs, and significantly outperforms state-of-the-art methods under stringent memory constraints.

This work addresses the joint optimization of media selection, capacity allocation, and data placement (replication vs. tiering) for key-value caching across heterogeneous NVM/DRAM/disk storage under memory budget constraints. We introduce the first systematic modeling framework for multi-level non-volatile cache configurations, analytically characterize the operational regimes where replication or tiering dominates, and propose an adaptive configuration policy grounded in device failure rates and data update frequencies. Our methodology integrates cache access behavior modeling, hierarchical configuration optimization, and empirical validation using memcached benchmarks. Results demonstrate that tiering substantially outperforms replication under low device failure rates and high update workloads. Key contributions include: (1) a deployable, low-overhead configuration algorithm; (2) quantitative design guidelines for heterogeneous cache deployment; and (3) theoretical foundations for the reliability–performance trade-off in tiered caching systems.

This work addresses the challenge in hierarchical caching networks where conventional request-time-based eviction policies fail to assess the impact of removing aligned storage blocks on the connectivity of downstream critical services, often causing service disruptions. The authors model aligned eviction as a weighted vertex separation problem on a graph, precisely computing the downstream demand cut of candidate blocks to reject evictions that compromise protected paths and selecting the feasible eviction with minimal impact. They introduce a novel verifiable service-cut certificate mechanism that, for the first time, unifies capacity reclamation, path continuity, and distributed failure within a certifiable interface, and prove that strategies relying solely on historical information can incur unbounded single-step damage. Experiments across 144 scenarios processing 582.9 trillion packets (404.86 PiB) validate theoretical predictions, reveal a zero-impact extremal phase transition point, and enable full impact vectors and audit samples within supplementary material budgets.

This study addresses a critical side-channel vulnerability in cloud environments where containers and virtual machines, despite employing software-based isolation mechanisms, remain susceptible to cross-tenant information leakage through the shared host page cache. The authors systematically evaluate the page cache risks across diverse runtime environments—including Docker, gVisor, Kata, and QEMU/KVM—under shared storage conditions, leveraging unprivileged timing measurements to infer cache residency across isolation boundaries. Their work is the first to demonstrate the pervasive nature of page cache leakage in modern isolation architectures and integrates this attack vector into a unified framework for OS-mediated microarchitectural timing side channels. Experiments confirm that the attack succeeds whenever the I/O path involves shared cacheable file objects, while mitigation strategies such as direct I/O or dedicated block devices significantly suppress the signal. The approach successfully recovers coarse-grained activity patterns from a real-world WordPress+MySQL deployment.

This work addresses the challenge of GPU memory inflation, request eviction, and throughput degradation in large language model serving under high concurrency, where growing key-value caches strain system resources. The authors develop a discrete-time dynamical model to characterize request admission, memory growth, and eviction mechanisms under continuous batching, revealing for the first time that service-induced congestion is a structurally unstable phenomenon. Theoretical analysis demonstrates that under homogeneous workloads, the no-eviction equilibrium is unstable, often driving the system toward a limit cycle with up to 50% throughput loss. In contrast, heterogeneous workloads with mutually coprime decoding lengths can achieve stable memory-constrained operation through desynchronization. Building on this insight, the study establishes stability criteria and scheduling design principles for multi-class requests, integrating methods from discrete dynamical systems, survival polynomials, and number theory.

This work addresses the significant memory and computational bottlenecks caused by KV caching in long-sequence inference, where existing eviction methods often degrade accuracy by inadvertently discarding critical information. The authors propose a training-free, magnitude-aware random eviction strategy, grounded in the novel observation that large-magnitude values are crucial for inference stability. By preferentially preserving high-magnitude states and incorporating randomness to enhance cache diversity, the method achieves robust performance while remaining compatible with FlashAttention-2 for static, memory-efficient inference. Evaluated across six reasoning tasks, the approach enables the Qwen3 model to attain an average accuracy surpassing current state-of-the-art sparse attention methods at a 4× KV cache compression ratio, outperforming the strongest eviction baseline by over 4%.

This work addresses the incompatibility between strict cache partitioning—effective against eviction-based side-channel attacks—and write-sharing coherence, which hinders its deployment in secure shared operating systems. The authors propose SCP, a novel mechanism that partitions only cache tags while maintaining a unified data pool, augmented with timing obfuscation and a last-level cache (LLC) write-routing policy. This approach uniquely reconciles strict eviction isolation with write-sharing coherence. SCP introduces leakage-threshold-driven dynamic writebacks and cross-partition lookup path protection, effectively neutralizing Prime+Probe, Flush+Reload, and shared writable-line attacks, reducing their success rates to random levels. Implemented in gem5, SCP incurs only a 2.8% increase in LLC SRAM overhead and less than 0.3% IPC performance degradation on SPEC CPU2017 benchmarks.