QEvict: Recoverable Quantized KV Eviction for Attention-Drift-Robust Long-Context Decoding

2026-08-05

AI Summary

This work addresses the irreversible nature of conventional KV cache eviction strategies, which are prone to permanently discarding critical historical states due to attention drift. To overcome this limitation, the authors propose a three-tier recoverable KV cache management mechanism that integrates dynamic cumulative attention scoring, quantization-dequantization operations, and window-level importance updates to retain broader contextual information under a fixed memory budget. By introducing diagnostic metrics such as Future Missed Mass and Global LIR, the method transcends the binary, irreversible constraints of traditional eviction schemes. Experimental results demonstrate that the proposed approach significantly outperforms existing eviction and quantization baselines on long-context understanding, retrieval, and reasoning tasks, effectively reducing attention omission and enhancing information retention.

Abstract

Autoregressive large language model inference is increasingly constrained by the memory footprint of the Key-Value (KV) cache. A dominant line of work reduces this footprint by evicting tokens that appear unimportant under attention-derived scores. However, such policies make an implicit irreversible decision: once a token is evicted, it cannot become useful again. We show that this assumption is brittle during decoding. Token and window importance drift as generated queries evolve, causing standard eviction policies to permanently discard states that later receive substantial attention under the full-cache model. To characterize this behaviour, we introduce Future Missed Mass and Global LIR, two diagnostics that measure future attention assigned to discarded states and the reactivation of historically inactive regions. We propose QEvict, a three-tier KV-cache management scheme that replaces binary retain-or-delete eviction with recoverable eviction. QEvict maintains high-confidence windows in full precision, stores intermediate windows in a quantized recoverable tier, and deletes only the lowest-confidence windows. During decoding, cumulative attention scores update window importance and when a quantized window becomes important again, it is dequantized and promoted to the full-precision. Under a fixed memory budget, this design preserves broader historical context while retaining exact full precision for the most important regions. Across long-context understanding, retrieval, and reasoning benchmarks, QEvict consistently improves over representative eviction and quantization baselines, reducing missed attention and improving information retention

Problem

• KV cache eviction
• attention drift
• recoverable quantization
• long-context decoding
• memory efficiency

Innovation

• recoverable eviction
• quantized KV cache
• attention drift
• three-tier caching
• Future Missed Mass