Hot-Cold Tiering of HBM and High Bandwidth Flash for Agentic LLM Serving

2026-09-22

AI Summary

为解决LLM服务中GPU内存容量限制导致的会话恢复成本问题，研究提出将活跃数据存于HBM、不活跃数据存于高带宽闪存的方法，形成热冷数据层次结构。

Abstract

Large language model (LLM) serving is increasingly agentic, with multi-turn sessions that idle between actions yet must retain their full context. Limited GPU memory capacity forces inactive KV states to be evicted, so resuming a session incurs either costly recomputation or slow interconnect transfers. To address this, high bandwidth flash (HBF)-an on-package 3D-NAND memory offering orders-of-magnitude greater capacity than high bandwidth memory (HBM) at comparable read bandwidth-has emerged as a strong candidate. However, its high read energy and limited write endurance make it impractical to serve all KV traffic. Fortunately, our analysis shows that agentic KV states exhibit distinct access patterns: a small hot set is read for every decoding step, while a large cold pool is read only when a paused session resumes. Exploiting this, we place the hot set in HBM and the cold pool in HBF, forming a hot-cold KV hierarchy within the GPU memory tier. On agentic workloads with Qwen3-Coder-30B-A3B, our design delivers 14 ms time-between-tokens (TBT) and adds only $\approx$0.1 ms of resume latency on top of prefill, while hosting $24\times$ more concurrent sessions per GPU. By confining HBM to the hot set, our design also cuts read power by 7.6 kW per 8-GPU node relative to serving all KV from flash-establishing HBF as a cold-tier complement to HBM rather than its replacement.

Problem

• Large language model (LLM) serving
• multi-turn sessions
• GPU memory capacity
• inactive KV states
• high bandwidth flash (HBF)

Innovation

• hot-cold tiering
• high bandwidth flash (HBF)
• key-value (KV) states
• GPU memory management
• agentic LLM serving