HiKV: Hierarchical Importance-Aware KV Cache with Hardware Acceleration for LLM Decoding

2026-07-24

AI Summary

This work addresses the memory bottleneck caused by the ever-growing KV cache in large language model decoding by proposing a two-stage, hierarchical, importance-aware compression approach through algorithm-hardware co-design. In the first stage, less important tokens are pruned under a fixed memory budget; in the second stage, only critical elements within the retained tokens are loaded. This is the first method to achieve dual-granularity KV cache compression, supported by a reconfigurable importance sorter that unifies the hardware acceleration paths for both stages. Under less than 1% accuracy degradation, the proposed design achieves up to 7.95× speedup and 90% energy reduction compared to the baseline. At comparable accuracy, it reduces external memory accesses by 1.82–4.87× with only an 8% increase in hardware area.

Abstract

With the rapid adoption of long-context large language models (LLMs), the continuously growing KV cache during decoding has become the critical memory bottleneck. To tackle this challenge, we propose HiKV, a novel algorithm-hardware co-design that exploits KV cache redundancy through hierarchical importance awareness. Algorithmically, HiKV compresses the KV cache at two granularities: Stage I evicts unimportant tokens within a fixed budget, and Stage II further loads only the significant elements of each retained token, reaching compression ratios unattainable at a single granularity. Architecturally, we develop a dedicated accelerator centered on a reconfigurable importance sorter that switches between the distinct sorting datapaths each stage requires, unifying the two-stage acceleration in one circuit with minimal overhead. Evaluated on representative LLMs, HiKV achieves up to 7.95x speedup and 90% energy reduction in the attention computation over the vanilla KV cache baseline within negligible 1% accuracy loss. Under iso-accuracy constraints, HiKV outperforms state-of-the-art importance-based methods by achieving an additional 1.82~4.87x reduction in external memory accesses. These benefits are enabled by specialized hardware components that add only 8% to the system area.

Problem

• KV cache
• memory bottleneck
• large language models
• decoding
• long-context

Innovation

• KV cache compression
• importance-aware pruning
• algorithm-hardware co-design
• reconfigurable accelerator
• LLM decoding