AnchorKV: Anchor-Residual KV Cache Compression

2026-08-03

AI Summary

This work addresses the memory bottleneck imposed by KV cache in large language models during long-context reasoning. The authors propose an anchor-residual compression mechanism that retains all input tokens without discarding any, yet stores only a small set of precisely selected anchor tokens explicitly; the remaining tokens are approximated via their most similar anchors, with residual refinements applied to critical tokens to preserve output fidelity. This approach achieves, for the first time, up to 20× compression of the KV cache, surpassing the accuracy-efficiency trade-offs inherent in conventional pruning and quantization methods. Experiments demonstrate that the method maintains 99% of the original performance on 70B-scale models while substantially reducing memory overhead, and it generalizes effectively across diverse architectures and datasets.

Abstract

The key-value (KV) cache is the primary memory bottleneck in long-context LLM inference. Existing approaches attack it from opposite ends: eviction methods permanently discard tokens, degrading performance whenever a discarded token later proves essential, while quantization methods retain all tokens at low precision but offer limited compression. We propose AnchorKV, a compression scheme that shrinks the cache by $20\times$ without discarding a single token. AnchorKV represents the cache using a small set of anchors stored exactly, expresses every other token through its most similar anchor, and refines only those whose approximation most affects the model's output. AnchorKV consistently preserves accuracy across models and datasets, retaining 99% of the full-cache score at the 70B scale, while keeping the entire context at a fraction of its cost.

Problem

• KV cache
• memory bottleneck
• long-context LLM inference
• cache compression

Innovation

• KV cache compression
• anchor-based representation
• residual refinement
• long-context LLM inference
• memory efficiency