Bridging LLM Serving and CXL-SSDs with Chunk-Aware KV Cache Management

2026-09-20

AI Summary

本文针对LLM前缀缓存的性能问题，提出了一种名为LM-CXD的CXL-SSD专用解决方案，通过优化KV块管理和预取策略显著提升了访问速度。

Abstract

NAND-backed storage offers the capacity needed to scale LLM prefix caching, but its block I/O path incurs CPU cache contention and host-DRAM staging in addition to NAND latency. Our characterization shows that these interface costs persist even with DRAM as the storage medium, motivating CXL-SSDs for byte-addressable access to NAND-backed capacity. Surprisingly, however, a stock CXL-SSD remains about 3$\times$ slower than local DRAM and no faster than an NVMe SSD, while generic prefetching provides little benefit. We present LM-CXD, a CXL-SSD specialized for LLM prefix caching. LM-CXD bridges the semantic gap between the serving engine, which knows which KV chunks will be consumed, and the device, which controls their placement and movement. It makes KV chunks device-visible I/O units, exposes NAND-to-DRAM progress to the serving engine, and uses device DRAM as a GPU-accessible buffer. LM-CXD further coordinates request scheduling with windowed prefetching and pipelines layerwise KV movement with GPU computation to hide NAND latency under limited device DRAM. Across five LLM models, LM-CXD reduces average TTFT over a stock CXL-SSD by up to 2.6$\times$ with compute asynchronous prefetching and 4.03$\times$ with layerwise prefetching, achieving TTFT within 1.5$\times$ of local DRAM on average.

Problem

• LLM prefix caching
• CXL-SSDs
• NAND-backed storage
• block I/O path
• prefetching

Innovation

• CXL-SSD
• chunk-aware KV cache management
• windowed prefetching
• layerwise prefetching