HyMCache: A KV Cache Framework for Multi-Turn LLM Serving with CXL-Hybrid Memory

2026-07-20

AI Summary

This work addresses the scalability limitations of KV cache reuse in multi-turn large language model (LLM) serving, which is constrained by the capacity and cost of GPU HBM and host DRAM, hindering terabyte-scale shared context support. To overcome this, the authors propose HyMCache, a novel framework that introduces CXL-based heterogeneous memory—combining a small amount of device DRAM with high-capacity SSDs—into multi-turn LLM inference. HyMCache enables efficient KV cache reuse through request-level prefix prefetching, opportunistic write buffering, and a reengineered DRAM management policy. Under identical DRAM budgets, HyMCache achieves 3.0× higher throughput per node than LMCache. Compared to Mooncake—a 1TB distributed DRAM solution—it reduces DRAM usage by 16× while incurring only ~30% performance degradation, effectively balancing high capacity, low cost, and low latency.

Abstract

Long-context, multi-turn, and agentic LLM workloads increasingly reuse previously processed context, making KV-cache reuse essential for reducing redundant computation. However, this reuse shifts the bottleneck to the memory tier that stores and serves reusable KV states at cluster scale. GPU HBM and host DRAM are too costly to scale to TB-scale shared context capacity, motivating remote tiers built from lower-cost, higher-capacity media. This paper presents HyMCache, a KV-cache framework that integrates CXL-hybrid memory (CXL-HM) for multi-turn LLM serving. CXL-HM combines a small amount of in-device DRAM with large SSD-backed capacity behind a CXL interface. By exploiting the read-dominant, predictable, and append-only nature of multi-turn KV-cache access, HyMCache rethinks DRAM management within CXL-HM to efficiently support TB-scale SSD-backed KV reuse. It uses request-level prefix prefetching and opportunistic write buffering to stage latency-critical reads in device DRAM, enabling DRAM-scale KV-cache efficiency at SSD-level cost. We evaluate HyMCache on a real CXL-HM prototype under both single-aggregator and PD-disaggregated serving configurations. Under the same DRAM budget, HyMCache outperforms local LMCache by 3.0x in single-node serving and 1.45x in PD-disaggregated serving. Compared with 1 TB distributed-DRAM Mooncake, HyMCache incurs about 30% lower performance but uses 16x less DRAM.

Problem

• KV cache
• multi-turn LLM serving
• CXL-hybrid memory
• memory bottleneck
• scalable context reuse

Innovation

• KV Cache
• CXL-Hybrid Memory
• Multi-turn LLM Serving
• Prefetching
• Memory Disaggregation