kv cache management

This work addresses the O(N²) computational bottleneck in long-context reasoning with large language models caused by standard attention mechanisms. The authors propose an asymmetric deep hashing-based approximate nearest neighbor attention mechanism that differentially encodes queries and keys, reframing attention computation as a hashing retrieval problem for the first time. By integrating a dynamic mixed-precision strategy—preserving full precision for critical tokens while using low-precision representations elsewhere—the method balances efficiency and generation quality. The resulting asymmetric KV-cache hashing framework substantially reduces memory and computational overhead, achieving performance on par with full attention on the LongBench benchmark while lowering inference complexity to O(N).

This work addresses the inefficiency of traditional batching strategies in large language model inference, which neglect request prefix homogeneity and consequently suffer from poor KV cache access patterns and limited throughput. The authors propose the first reinforcement learning–based, prefix-aware scheduler that dynamically balances batch size against prefix sharing. To enable efficient prefix matching and request selection, they introduce a novel Chunked Hash Tree data structure, significantly enhancing both spatial and temporal locality of KV cache accesses. Integrated into vLLM and SGLang inference engines, the method achieves 2–10× higher end-to-end throughput over existing schedulers on workloads with shared prefixes, while maintaining baseline-level performance in scenarios without prefix sharing.

This work addresses the substantial memory and computational overhead imposed by KV caching in long-context reasoning, where existing pruning methods often degrade model performance. The authors propose a learnable KV cache pruning approach featuring a global retention mechanism, reframing pruning not merely as compression but as a means to enhance inference quality. By employing a lightweight retention gate to predict the future utility of tokens—augmented with query-agnostic geometric retention proxies, shared scoring projections, and a global competition strategy across layers, heads, and modalities—the method effectively mitigates attention dilution under a unified memory budget. Experiments demonstrate that the approach significantly reduces KV cache consumption while matching or even surpassing the performance of full caching across diverse benchmarks, including long-context language modeling, vision-language tasks, and multi-turn dialogue.

This work addresses the inefficiency of standard key-value (KV) caching in large language models, where context dependence necessitates recomputation when reusing cached states in new contexts, incurring significant computational overhead and latency. To overcome this limitation, the authors propose the KV Packet framework, which encapsulates cached KV states into immutable “packets” and introduces a lightweight, trainable soft token adapter to bridge contextual discontinuities. This approach enables, for the first time, fully recomputation-free, context-agnostic reuse of KV caches. Combined with a self-supervised distillation strategy to correct attention distribution shifts, the method achieves near-zero FLOPs overhead and reduced first-token latency on Llama-3.1 and Qwen2.5, while maintaining F1 scores comparable to those of full recomputation baselines.

To address the severe non-linear scaling of prefill latency with context length—particularly prohibitive for million-token contexts in large language model inference—this paper introduces a context-parallel architecture augmented with lossless ring attention variants (pass-KV/pass-Q), enabling unified support for full prefill, persistent KV prefill, and decoding. The method integrates cross-GPU/cross-node context partitioning, RDMA/TCP-hybrid communication scheduling, and multi-node distributed tensor parallelism on H100 clusters. Evaluated on Llama3-405B, it achieves 77 seconds for 1M-token prefill (93% parallel efficiency) and 3.8 seconds for 128K tokens, maintaining strong scalability even in commodity data centers with moderate bandwidth. The core contribution is a novel long-context inference paradigm that simultaneously ensures numerical fidelity, operational generality across inference modes, and near-linear scalability.

This work addresses the challenge of achieving efficient and accurate reasoning over extremely long contexts without modifying or fine-tuning the underlying model. It proposes a training-free chunked recurrence protocol that, for the first time, reveals frozen pre-trained Transformers inherently support a stable key-value (KV) cache recurrence mechanism. By treating the KV cache as a cross-chunk accumulator and employing single-step recurrence combined with cache concatenation primitives, the method enables inter-segment information propagation while preserving numerical stability and memory efficiency through reuse of internal model states. Experiments demonstrate that Llama-3.1-8B achieves 100% exact retrieval accuracy across context lengths from 16K to 128K, supports reasoning chains up to depth 511, and operates on a single 40GB GPU.

This work addresses the high memory demand of key-value (KV) caching in long-context large language model inference, which often leads to GPU memory overflow. Existing offloading approaches suffer from excessive data transfer and decoding latency under long-context and large-batch settings. To overcome these limitations, the authors propose a multi-tier KV cache management system spanning GPU memory, host DRAM, and SSD. The system employs an attention-aware caching policy to dynamically enhance cache reuse, restructures the decoding pipeline to overlap I/O with computation, and coordinates cross-tier data migration in a unified manner. Experimental results on mainstream large language models and long-context benchmarks demonstrate up to a 1.74× throughput improvement over prior methods, without compromising model accuracy.

This work reveals a critical security vulnerability in position-agnostic key-value (KV) cache reuse mechanisms, wherein an adversary can hijack model behavior by manipulating cached context—even when the input itself contains no malicious content. To systematically exploit this flaw, the paper introduces HIJACKKV, a novel attack framework that optimizes an attacker-controlled prefix so that the KV cache associated with benign text implicitly encodes a malicious target. During cache reuse, this enables stealthy control over model outputs. The method achieves an average success rate of 94% in single-shot attacks and remains effective under challenging conditions, including low cache hit rates (10%) and high recomputation rates (50%). Furthermore, it demonstrates cross-model black-box transferability and offers practical recommendations for designing secure caching mechanisms.

This work addresses the escalating inference costs in long-horizon LLM agent execution caused by accumulating context, where existing pruning or memory eviction strategies often disrupt prompt cache continuity, leading to prefix mismatches and cache invalidation. To tackle this, the authors propose TokenPilot, a dual-granularity context management framework that globally employs ingestion-aware compression to stabilize the prompt prefix and filter environmental noise, while locally applying a lifetime-aware eviction mechanism to conservatively offload context once its task relevance diminishes. TokenPilot is the first approach to achieve efficient context compression without compromising cache continuity, effectively balancing textual sparsity and cache efficacy. Experiments on PinchBench and Claw-Eval demonstrate that TokenPilot reduces inference costs by up to 61%/56% in isolated mode and 61%/87% in continuous mode, with no degradation in task performance, and has been integrated into LightMem2.

This work addresses the performance degradation in agentic LLM reasoning caused by token position shifts, which invalidate prefix caching and lead to significant increases in time-to-first-token (TTFT) latency. To overcome this limitation, the study introduces a novel position-agnostic cache reuse mechanism by natively integrating content-addressable caching into the Multi-head Latent Attention (MLA) architecture. The proposed approach leverages content-based hashing, CDC chunking, and δ-rotation rules to manage low-dimensional rotary key vectors in MLA, eliminating reliance on conventional prefix matching. Experiments across three MLA-MoE models demonstrate that, upon cache hits, the method recovers up to 83% of prompt tokens, reduces prefill energy consumption by 63%, and strictly preserves output consistency.