C$^2$KV: Compressed and Composable KV Cache Reuse for Efficient LLM Inference

2026-07-20

AI Summary

This work addresses the substantial storage and access overhead of key-value (KV) caches in long-context large language model inference, where existing reuse methods struggle to balance efficiency and accuracy. The authors propose C²KV, a unified framework that enables modular KV cache reuse through position-agnostic, composable compressed KV representations. C²KV jointly optimizes extraction and inference-time concatenation via a lightweight extractor, learnable compression tokens, and structured attention flows, all trained under a compression-concatenation co-training strategy while keeping the base model frozen. Experimental results demonstrate that C²KV reduces KV cache memory consumption by up to 17× and significantly accelerates inference across multiple long-context benchmarks, all while preserving generation quality.

Abstract

Long-context inference is central to modern large language model (LLM) applications such as retrieval-augmented generation and multi-document reasoning. To mitigate the growing inference cost, recent work has explored key-value (KV) cache reuse to reduce redundant prefill computation. However, existing reuse methods primarily focus on computation savings and overlook a critical bottleneck in long-context LLM serving: the cost of storing and accessing large KV caches. While KV compression appears to be a natural complement, naively combining compression with non-prefix KV reuse often leads to severe accuracy degradation. In this work, we propose C$^2$KV, a unified framework for non-prefix KV reuse that jointly optimizes KV extraction and inference-time concatenation. C$^2$KV learns a composable and compressed KV cache manifold that is explicitly designed to be position-agnostic. Our approach introduces a lightweight sidecar Extractor with learnable compression tokens and a structured attention flow, enabling modular KV representations that can be flexibly reused and concatenated without modifying the frozen base model. We further employ a compression-concatenation co-training strategy to align extraction-time representations with their downstream reuse behavior. Extensive experiments across multiple long-context benchmarks and model families demonstrate that C$^2$KV significantly reduces KV cache storage and transfer costs, achieving up to 17$\times$ inference speedup under long contexts, while preserving generation quality.

Problem

• KV cache reuse
• long-context inference
• KV compression
• storage bottleneck
• accuracy degradation

Innovation

• KV cache reuse
• compression
• composable representation
• long-context LLM inference
• structured attention