Problem
Research questions and friction points this paper is trying to address.
KV cache compression
long-context LLMs
eviction timing
memory overhead
one-shot compression
DeferKV: Rethinking Eviction Timing for One-Shot KV Cache Compression
🤖 AI Summary
This study addresses the substantial memory overhead of KV caches in long-context large language models, where existing methods perform irreversible eviction prematurely before generation signals emerge. To overcome this limitation, this work proposes a novel paradigm that defers eviction decisions until the first decoding step. By integrating attention signals from both the prefill and early decoding phases, the approach accurately evaluates KV importance to align with subsequent generation requirements, enabling unsupervised dynamic compression without additional training or auxiliary modules. This contribution breaks through the conventional framework of immediate post-prefill eviction, significantly enhancing model performance under high compression ratios on benchmarks such as LongBench while maintaining low inference latency.
📝 Abstract
Long-context large language models (LLMs) have demonstrated strong capabilities across a wide range of tasks, but the growing KV cache introduces substantial memory and inference overhead. Existing one-shot KV cache compression methods typically commit to irreversible eviction immediately after prefill, before any signal from actual generation becomes available. Our quantitative analysis shows that early queries from the actual generation stage provide attention signals that are more consistent with subsequent decode attention, with the largest single-step gain occurring at the prefill-decode boundary. Based on this observation, we propose DeferKV, which moves the eviction decision from the end of prefill to the first real decoding step and temporally combines prompt-side and decode-side observations, thereby better aligning KV importance estimation with subsequent generation requirements. DeferKV requires no additional training, draft model, or future-query prediction module, making it simple and easy to deploy. Experiments on LongBench, RULER, and Needle-in-a-Haystack demonstrate that DeferKV consistently improves model performance under KV cache compression while maintaining low inference latency.
Innovation
Methods, ideas, or system contributions that make the work stand out.
KV Cache Compression
Eviction Timing
Long-context LLMs
One-Shot Compression
Attention Signal
🔎 Similar Papers
No similar papers found.
