DualDecoder: Accelerate Long Context LLM Inference by Predictive Prefetch

2026-07-29

AI Summary

This work addresses the severe memory bottleneck in long-context large language model inference, where KV cache size grows linearly with both context length and concurrent request count. Existing sparse KV caching methods introduce substantial auxiliary GPU state, creating new bottlenecks under high concurrency. To overcome this, the authors propose DualDecoder, a novel system that leverages speculative token prediction to identify critical KV positions needed for the next decoding step. By integrating a dual-token decoding pipeline, layer-aware transfer scheduling, and layer-wise GPU memory management, DualDecoder enables predictive prefetching with zero additional GPU memory overhead, effectively overlapping KV retrieval and computation. The system achieves up to 2.62× higher throughput while preserving model quality and latency.

Abstract

Long-context inference is becoming a fundamental capability for modern LLM serving, especially driven by emerging agentic applications. Yet it faces a severe memory wall that the KV cache scales proportionally with increasing context length and request concurrency. Existing sparse KV cache methods offload most KV entries to host memory and retrieve only the critical KV entries needed by each decoding step. However, they commonly introduce substantial auxiliary states in GPU memory for KV retrieval management. Our measurements show that these often-overlooked auxiliary states introduce significant memory overhead and become a new bottleneck under high-concurrency workloads. In this paper, we present DualDecoder, a lightweight serving system for long-context LLM inference that enables efficient sparse KV cache retrieval from host memory. Our key insight is that the critical KV entries required for decoding the next token can be accurately predicted from the preceding speculated token. This predictability enables KV retrieval to be proactively prefetched and overlapped with decoding computation, effectively eliminating the GPU memory overhead of auxiliary states. To achieve this prefetching efficiently, DualDecoder leverages a novel dual-token decoding pipeline that accurately identifies critical KV entries with negligible computational overhead, and designs a layer-aware transfer schedule to overlap KV prefetching with model computation and a layer-scoped memory manager to reduce the GPU runtime buffer. Experimental results show that DualDecoder improves decoding throughput by up to 2.62$\times$ over state-of-the-art systems while preserving decoding latency and model quality.

Problem

• long-context inference
• KV cache
• memory overhead
• high-concurrency
• LLM serving

Innovation

• DualDecoder
• sparse KV cache
• predictive prefetching
• long-context LLM inference
• memory-efficient serving