SparseEngine: Sparse-First Inference Engine

2026-09-30

AI Summary

This study addresses the KV cache memory and attention computation bottlenecks in long-context LLM agents, as well as the integration challenges of existing sparsity methods, by proposing a sparsity-first inference engine. The engine coordinates heterogeneous cache representations and serving infrastructure through a unified lifecycle contract, and introduces a chained caching mechanism for cross-request state management that supports controllable prefix cache pruning to preserve logical matching. The system is compatible with fifteen sparse attention methods. Experimental results demonstrate that the proposed approach achieves over a 10× throughput improvement, a 2.5× faster decoding speed compared to vLLM, and more than a 2× end-to-end acceleration.

Abstract

Long-context LLM agents accumulate interaction histories that strain KV-cache memory and attention computation. Although sparse attention reduces these costs, heterogeneous cache representations and workflows hinder integration with existing inference engines, while prior sparse-serving abstractions support only specific layouts or workflows. We present SparseEngine, a ground-up, sparse-first inference engine whose shared lifecycle contract lets each method control its KV representation and computation while coordinating state transitions with common serving infrastructure. SparseEngine supports 15 methods across four categories and enables cross-request state management through Chain Cache, which resumes KV-eviction methods from retained history, and controllable Prefix-Cache Pruning, which removes KV from selected history regions while preserving logical-prefix matching. While maintaining method quality, SparseEngine delivers over 10x higher throughput with KV eviction, over 2.5x faster decoding at matched concurrency than vLLM, and over 2x end-to-end speedup on agent benchmarks. The code is available at https://github.com/CURRENTF/SparseEngine.

Problem

• long-context LLM
• KV-cache
• sparse attention
• inference engine
• heterogeneous cache

Innovation

• Sparse Attention
• Inference Engine
• KV Cache Eviction
• Chain Cache
• Long-context LLM