Value-Aware Stochastic KV Cache Eviction for Reasoning Models

2026-06-02

AI Summary

This work addresses the significant memory and computational bottlenecks caused by KV caching in long-sequence inference, where existing eviction methods often degrade accuracy by inadvertently discarding critical information. The authors propose a training-free, magnitude-aware random eviction strategy, grounded in the novel observation that large-magnitude values are crucial for inference stability. By preferentially preserving high-magnitude states and incorporating randomness to enhance cache diversity, the method achieves robust performance while remaining compatible with FlashAttention-2 for static, memory-efficient inference. Evaluated across six reasoning tasks, the approach enables the Qwen3 model to attain an average accuracy surpassing current state-of-the-art sparse attention methods at a 4× KV cache compression ratio, outperforming the strongest eviction baseline by over 4%.

Abstract

Reasoning models improve accuracy through extended chains of thought, but their long outputs create a memory and compute bottleneck. KV cache eviction methods reduce this cost by evicting unimportant key-value pairs from the cache, yet they often yield worse accuracy than selection-based sparse attention alternatives, which keep the full KV cache. We identify key factors crucial to KV cache eviction accuracy. First, a small fraction of value states have abnormally large magnitudes, and evicting them causes catastrophic failure where models enter repetitive reasoning loops. Second, introducing stochasticity during eviction improves accuracy by increasing cache diversity. Based on these findings, we propose Value-aware Stochastic KV Cache Eviction (VaSE), a training-free recipe that protects large-magnitude value states and promotes diverse eviction decisions. Across six reasoning tasks, Qwen3 models using VaSE with 4x KV cache compression yield higher average accuracies than SOTA selection method at the same sparsity, while outperforming the strongest eviction method by more than 4%. Overall, VaSE bridges the gap between efficiency and accuracy, supporting FlashAttention2 and enabling a static memory footprint for reasoning models.

Problem

• KV cache eviction
• reasoning models
• memory bottleneck
• accuracy degradation
• efficient inference

Innovation

• KV cache eviction
• value-aware
• stochastic eviction
• reasoning models
• memory efficiency