Every Cache Entry Earns Its Place: Global Allocation of Resolution and Coverage for KV Cache Compression

2026-08-07

AI Summary

This work addresses the limitations of large language models in long-context reasoning caused by the storage and access overhead of KV caches. Existing compression methods, relying on fixed rules, struggle to flexibly balance local resolution and global information coverage across layers, heads, and token positions. The paper formulates KV cache compression for the first time as a global resource allocation problem under a fixed budget and introduces a prototype-tree-based dynamic optimization framework. Treating non-overlapping nodes as atomic units, this framework adaptively trades off between expanding coverage and enhancing resolution, enabling fine-grained cache competition across layers, heads, and positions. Requiring no additional training and supporting online GPU inference, the method outperforms existing approaches in 24 out of 32 evaluated long-context tasks and compression ratios, maintaining robust performance even at up to 128× compression.

Abstract

As large language models (LLMs) process increasingly long contexts, KV cache storage and repeated access have become a major bottleneck. Existing KV cache compression methods rely on predefined, fixed compression rules and are typically developed around either token eviction or merging. As a result, cache resources can neither flow freely across layers, heads, and context slots, nor be jointly allocated to balance local resolution and information coverage. Therefore, we propose GraceKV, a global approach for the allocation of resolution and coverage in KV cache compression, and formulate the compression process as a global resource allocation problem under a fixed cache budget. GraceKV treats each layer-KV head-slot combination as an atomic unit and builds a prototype tree. Leaf nodes correspond to token-level KV entries, while each internal node uses a single prototype to compress the KV space covered by its children. A set of non-overlapping nodes in the tree forms the representation of an atomic unit. Adding the root of a new tree expands information coverage, whereas splitting a selected node improves local resolution. All candidate actions compete globally for a shared cache budget. Finally, the nodes retained across all trees form the compressed KV cache. This process adaptively determines the allocation of cache resources among atomic units globally and the balance between resolution and coverage. GraceKV requires no additional training, and the entire compression and inference process is performed on the GPU. Systematic experiments across diverse long-context tasks and compression ratios show that GraceKV ranks first in 24 of 32 settings and remains robust up to 128-fold compression. These results validate the effectiveness of global budget allocation in coordinating information coverage and local resolution.

Problem

• KV cache compression
• resource allocation
• information coverage
• local resolution
• long-context LLMs

Innovation

• KV cache compression
• global resource allocation
• prototype tree
• resolution-coverage trade-off
• long-context LLMs