Key, Value, Compress: A Systematic Exploration of KV Cache Compression Techniques

2025-03-14

AI Summary

Large language models (LLMs) suffer from quadratic attention complexity and explosive KV cache memory consumption and latency when processing long contexts (≥32K tokens). To address this, we propose the first comprehensive methodology classification framework for KV cache compression, unifying key techniques—including quantization, pruning, low-rank approximation, sequence grouping, and locality-aware reuse—across both theoretical principles and practical implementation dimensions. Through standardized cross-method evaluation, we characterize the fundamental trade-offs among accuracy, inference speed, and memory footprint. Experimental results demonstrate that, with perplexity degradation under 1%, the optimal compression strategy achieves 2–5× KV cache memory reduction and 1.3–2.1× inference speedup, substantially enhancing the feasibility of deploying LLMs in long-context scenarios.

Technology Category

Machine Learning: Large Multimodal Models (LMMs)Natural Language Processing: (Large) Language ModelsData Mining & Knowledge Management: Data Compression

Application Category

Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMsUser Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendationSearch and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Large language models (LLMs) have demonstrated exceptional capabilities in generating text, images, and video content. However, as context length grows, the computational cost of attention increases quadratically with the number of tokens, presenting significant efficiency challenges. This paper presents an analysis of various Key-Value (KV) cache compression strategies, offering a comprehensive taxonomy that categorizes these methods by their underlying principles and implementation techniques. Furthermore, we evaluate their impact on performance and inference latency, providing critical insights into their effectiveness. Our findings highlight the trade-offs involved in KV cache compression and its influence on handling long-context scenarios, paving the way for more efficient LLM implementations.

Problem

• Addresses computational cost in large language models
• Explores KV cache compression for efficiency
• Evaluates impact on performance and latency

Innovation

• Explores KV cache compression techniques systematically
• Categorizes methods by principles and implementation
• Evaluates impact on performance and latency