ResKV: Reconstructing Omitted Attention Contributions for Fixed-Budget KV Cache Compression

2026-07-31

AI Summary

This work addresses a critical limitation in existing KV cache compression methods, which permanently discard the attention contributions of pruned tokens, thereby degrading model performance. To mitigate this issue, the authors propose a residual caching mechanism that partitions a fixed KV cache budget into a precise primary cache and a compact residual cache. This design enables joint reconstruction of the full attention contributions—both numerator and denominator—of discarded tokens within the softmax normalization. Leveraging statistical modeling of attention residuals, a hierarchical residual allocation strategy, and a dynamic query gating mechanism, the method optimizes residual distribution without perturbing the primary cache. Experiments on LongBench and RULER benchmarks demonstrate that the proposed approach significantly outperforms current compression techniques, enhancing long-context reasoning under identical KV budgets while preserving memory efficiency and decoding throughput.

Abstract

KV cache compression is essential for efficient long-context inference. Existing eviction methods permanently discard unselected tokens and consequently remove their aggregate contribution to attention. Merging-based alternatives preserve more information but can perturb retained keys and values that should remain exact. We observe that the information omitted by cache eviction can be formulated as residual statistics in both the numerator and denominator of softmax attention. Based on this observation, we propose ResKV, which divides a fixed KV budget into an exact main cache and a compact residual cache that reconstructs the contribution of omitted tokens. ResKV lets main-cache tokens and residual entries participate in the same softmax normalization, so residual entries restore both attention numerator and denominator mass rather than acting as a post-hoc correction. A construction-time validation proxy determines residual allocation for each layer and KV head, while a decode-time dynamic gate adjusts residual contributions for individual queries. Comprehensive evaluations on LongBench and RULER, covering query-aware and query-agnostic settings, multiple backbones, cache budgets, and representative compression baselines, demonstrate broad improvements under the same retained KV budget while preserving the practical efficiency of compressed decoding, including peak memory usage and long-context decode throughput.

Problem

• KV cache compression
• attention mechanism
• fixed-budget
• information omission
• long-context inference

Innovation

• KV cache compression
• attention residual
• softmax normalization
• fixed-budget inference
• long-context LLM