EchoPress: Query-Agnostic KV Cache Pruning via Virtual Context Reconstruction

2026-09-30

AI Summary

This study addresses the high overhead of KV cache pruning in long-context reasoning, which typically relies on additional forward passes or specialized training. We propose EchoPress, a training-free method that approximates attention scores using queries and keys from the standard prefill stage to identify critical KV pairs. Furthermore, it introduces a query-agnostic virtual context reconstruction mechanism requiring only first-chunk calibration to efficiently evaluate the importance of the remaining context. Experimental results demonstrate that EchoPress achieves accuracy comparable to KVzip on benchmarks such as LongBench while reducing compression overhead by 1.7× to 19.6× and decreasing total prefill time by nearly 3×.

Abstract

KV cache pruning reduces long-context inference memory usage by evicting less important key-value pairs. KVzip estimates importance through context reconstruction: prompting a model to repeat the context chunk by chunk. This achieves strong compression quality at the cost of additional forward passes. Learned approximations reduce this cost but require model-specific training. We analyze how KVzip identifies important cached information and show how to approximate its reconstruction scores using information already computed during prefill. These findings motivate EchoPress, a training-free method that approximates reconstruction attention using queries and keys from standard prefill. For each request, it reconstructs only the first chunk to calibrate importance scores for the remaining context. Experiments on LongBench and RULER with Qwen3-8B and Llama-3.1-8B-Instruct show that EchoPress matches KVzip in task accuracy across eviction ratios from 50% to 90%, while reducing compression overhead by a factor of 1.7-19.6 and total prefill time by a factor of up to 2.9. Code is available at https://github.com/ljwljwljwljw/kvpress/tree/echo-press.

Problem

• KV cache pruning
• long-context inference
• memory efficiency
• context reconstruction
• training-free

Innovation

• KV cache pruning
• virtual context reconstruction
• training-free
• query-agnostic
• prefill approximation