S$^4$R: Selective Sampling, Subspaces, and Sparse Reconstruction for Compressed Long-Context KV Caching

2026-08-01

AI Summary

This work addresses the challenge of excessive memory overhead from key-value (KV) caching in large language models under long-context scenarios, where existing low-rank compression methods struggle to balance reliance on offline calibration and online computational efficiency. The authors propose a calibration-free, efficient compression approach that constructs a low-rank subspace via prompt-aware initialization and dynamically adapts to input prompts during decoding through selective sampling and sparse reconstruction. This method substantially reduces KV cache memory usage while preserving high model accuracy. Experimental results on the LongBench and RULER benchmarks demonstrate that, when applied to Llama and Qwen architectures, the technique achieves up to 5× KV cache compression with negligible degradation in performance compared to using the full cache.

Abstract

The growth of context window lengths in Large Language Models (LLMs) significantly enhances their long-context capabilities but incurs prohibitive memory costs due to the Key-Value (KV) cache. Although low-rank compression of KV cache is a promising remedy, existing methods face a dilemma: offline approaches depend on external calibration data, whereas online approaches incur substantial compute for full-prompt decomposition and reconstruction. In this paper, we propose S$^4$R, which builds low-rank subspaces from selectively sampled tokens and computes attention over a sparsely reconstructed KV representation. S$^4$R uses prompt-aware initialization to build initial key/value bases from a representative prompt subset, trading off calibration-data dependence against prefilling cost. Because fully reconstructing the cache at every decoding step is prohibitively expensive and hurts throughput, we further adopt sparse reconstruction to retain only informative positions during decoding. Extensive experiments on LongBench and RULER with Llama and Qwen model families show that S$^4$R achieves up to 5$\times$ KV compression with near full-cache accuracy, combining the efficiency of fixed compression with the adaptability of prompt-dependent methods.

Problem

• KV caching
• long-context
• memory efficiency
• low-rank compression
• Large Language Models

Innovation

• Selective Sampling
• Low-Rank Subspaces
• Sparse Reconstruction
• KV Cache Compression
• Long-Context LLMs