HiSparse: Scaling Sparse-Attention Decoding with Hierarchical KV Cache Management

2026-08-07

AI Summary

This work addresses the memory bottleneck in long-context large language model inference, where maintaining the full KV cache on GPU limits support for ultra-long sequences. The authors propose an indexer-agnostic hierarchical KV caching architecture that keeps the complete KV history in host memory while retaining only a fixed-size working set on the GPU. A fused CUDA kernel efficiently handles cache hit detection, eviction, and data migration in a single pass. For models with cross-layer shared attention heads, they introduce a precise per-layer prefetching mechanism that hides nearly half of the miss penalty. Evaluated on H200, B200, and GH200 systems, the approach achieves up to 4.7× higher throughput and significantly reduced first-token latency, without introducing additional computational overhead or altering model outputs.

Abstract

Top-k sparse attention makes long-context LLM decoding cheap to compute: each step reads only a few thousand selected KV entries rather than the full context. Serving systems, however, typically keep the entire KV cache in GPU HBM so that every position stays selectable, so a request's memory bill still grows with its full context length--decoding hits a capacity wall long before it runs out of compute, and a context whose KV cache exceeds HBM cannot be served at all. We present HiSparse, an exact, indexer-agnostic hierarchical KV cache for sparse-attention serving. HiSparse keeps each request's full KV history in host memory and bounds its decode footprint with a small, fixed-size GPU cache; a fused CUDA kernel resolves each layer's selections--hit detection, LRU replacement, and host-to-device fetches--inside the decode CUDA graph; and, for models that share selections across layers, exact layer-wise prefetching hides roughly half of the remaining miss overhead. Because only KV placement changes, model outputs are unchanged. HiSparse is merged into upstream SGLang and evaluated across three sparse-attention families (DSA, NSA, and Quest) on H200, B200, and GH200 platforms: it improves peak generation throughput by up to 4.7x on long-context workloads while preserving comparable per-token latency and reducing time-to-first-token at high load--and a no-IO oracle shows the resolution mechanism itself adds no measurable per-token cost, leaving host-device IO as the only price of bounded residency.

Problem

• sparse attention
• KV cache
• memory bottleneck
• long-context decoding
• LLM serving

Innovation

• sparse attention
• KV cache management
• hierarchical memory
• CUDA kernel fusion
• prefetching