SmartGen: Seamless Disaggregated LLM Inference with Selective KV Cache Transfer

2026-07-30

AI Summary

This work addresses the substantial inter-node communication overhead in decoupled large language model (LLM) inference within cloud environments, where separating prefill and decode stages incurs high bandwidth costs due to full KV cache transmission. To mitigate this, the authors propose a selective KV cache transfer mechanism that transmits only critical cache entries. The approach leverages profiling-driven importance prediction and integrates three data pathways: proactive transfer, parallel on-demand fetching, and speculative completion. As the first selective KV transmission scheme tailored for decoupled LLM inference, it achieves up to a 4.3× reduction in time-to-second-token latency while preserving generation accuracy and decode throughput, significantly alleviating cross-node communication demands.

Abstract

Disaggregating the prefill and decoding stages of large language model (LLM) inference into two separate sets of nodes is widely adopted in today's LLM serving systems. However, such an architecture poses significant challenges for self-hosted LLM deployments on rented cloud instances, since transferring enormous key-value (KV) caches between disaggregated nodes can easily saturate the limited inter-node network bandwidth. In this paper, we propose to mitigate the network bottleneck by selectively transferring essential KV cache entries across the two stages. There are two challenges to achieve selective KV cache transfer, i.e., accurate KV selection during the prefill stage, and efficient KV fetching during the decoding stage. To address these challenges, we design SmartGen, a KV cache transfer engine that allows seamless disaggregated LLM inference with three data transfer paths. Specifically, we leverage 1) a profile-based proactive transfer path to identify and push essential KV cache entries to the decoding node during the prefill stage, 2) a parallel on-demand transfer path to simultaneously fetch remote and local KV cache entries during the decoding stage, and 3) a speculative transfer path to finally deliver all KV caches to the decoding node. Experimental results show that SmartGen reduces time-to-second-token by up to 4.3x compared with the typical full KV cache transfer approach while offering comparable subsequent decoding performance and accuracy.

Problem

• disaggregated LLM inference
• KV cache transfer
• network bandwidth bottleneck
• cloud-based LLM serving
• self-hosted LLM deployment

Innovation

• Selective KV Cache Transfer
• Disaggregated LLM Inference
• SmartGen
• Prefill-Decoding Disaggregation
• Network Bandwidth Optimization