VarRate: Training-Free Variable-Rate KV Cache Compression for Long-Context LLMs

2026-07-16

AI Summary

This work addresses the memory bottleneck caused by KV cache in long-context large language model inference, particularly the significant accuracy degradation of existing training-free compression methods in query-agnostic reuse scenarios. To this end, we propose a training-free, variable-rank KV cache compression method that dynamically allocates a low-rank budget to each token based on its query saliency. By preserving all tokens and avoiding irreversible pruning, our approach mitigates accuracy collapse. Experiments show that with only 20% of the original cache budget, our method achieves an average performance drop of just 0.8 points on LongBench compared to the uncompressed model. It consistently attains state-of-the-art compression efficiency under identical memory constraints on both Llama-3.1-8B and Qwen2.5-7B, while incurring only approximately one-eighth of the prefill overhead of KVzip.

Abstract

The key-value (KV) cache is the main memory bottleneck in long-context large language model (LLM) inference. Two leading training-free families are both structurally limited: token-selection methods (SnapKV, Ada-KV) score importance from an observation window and evict low-scoring tokens, but eviction is irreversible -- so when the importance signal degrades under query-agnostic reuse, accuracy collapses by 11-15 points; uniform low-rank coding keeps every token but spends equal rank everywhere, wasting budget. We observe that both failures share one cure: rank should be allocated, not evicted. We present VarRate, a training-free KV codec that assigns each token a variable low-rank budget by its query salience, keeping every token at a nonzero rank. Comparable adaptive-rank codecs reach this allocation only through training; VarRate requires none. Because no token is dropped, it degrades by only 3.5-5.5 points where query-aware selection collapses. At a matched 20% budget on LongBench (16 tasks), VarRate stays within 0.8 points of the uncompressed model on both Llama-3.1-8B and Qwen2.5-7B. Averaged over the two, it is the strongest matched-memory compressor. It significantly beats its uniform-rank ablation on both models. Against KVzip, a method purpose-built for query-agnostic reuse, it is accuracy-equivalent in three of four settings and within a point overall, at about one-eighth the prefill overhead.

Problem

• KV cache compression
• long-context LLMs
• memory bottleneck
• training-free compression
• variable-rate allocation

Innovation

• variable-rate compression
• KV cache
• training-free
• adaptive rank allocation
• long-context LLMs