Problem
Research questions and friction points this paper is trying to address.
KV cache compression
long-context inference
query-agnostic pruning
large language models
reusable cache
TaskPress: Query-Agnostic KV Cache Compression via Task-Guided Pruning
🤖 AI Summary
This work addresses the challenge that large language models face in long-context reasoning due to the linear growth of key-value (KV) cache with sequence length, compounded by the query-specific nature of existing pruning methods that hinders cross-query reuse. To overcome this, the authors propose TaskPress, a novel framework that achieves task-oriented, query-agnostic KV cache compression for the first time. TaskPress introduces high-level task instructions as meta-queries during the prefilling phase to proactively filter irrelevant tokens and construct a compact, reusable cache. It further leverages quantization scaling factors as a zero-overhead proxy for token importance. Experimental results demonstrate that TaskPress substantially reduces memory overhead across diverse long-context tasks while preserving model performance.
📝 Abstract
Long-context inference with large language models is constrained by the linear growth of the key-value cache to sequence length. While pruning offers mitigation, prevailing methods determine query-specific token importance that cannot be reused across unseen queries. In contrast, we introduce TaskPress, a framework for task-guided, query-agnostic KV cache eviction. Instead of optimizing the cache for a single query, TaskPress constructs a reusable memory representation conditioned on a high-level task guide. The guide functions as a meta-query during prefill to filter irrelevant tokens before downstream queries are issued. In addition, TaskPress leverages quantization scale factors as a zero-cost signal for detecting influential representation outliers, providing an efficient proxy for token importance. Experiments on conducted on various tasks with long context input demonstrate that TaskPress efficiently creates a compact, reusable cache across diverse queries.
Innovation
Methods, ideas, or system contributions that make the work stand out.
KV cache compression
query-agnostic pruning
task-guided memory
quantization scale factors
reusable cache
🔎 Similar Papers
No similar papers found.
