Lachesis: Lifetime-Aware KV Cache Placement for Agent Serving across HBM and High-Bandwidth Flash

2026-10-06

AI Summary

This study addresses the challenges of high KV cache memory consumption and limited flash write endurance in agent-based services by proposing a lifetime-aware hierarchical storage architecture. To our knowledge, this work is the first to leverage orchestrator behavior analysis for predicting cache lifetimes, dynamically retaining short-lived data in High Bandwidth Memory (HBM) while offloading long-lived data to flash storage, thereby enabling coordinated scheduling across heterogeneous media. Trace-driven simulation results demonstrate that the proposed approach extends flash lifespan by 1.19× to 3.13× compared to conventional methods. Furthermore, under full-load scenarios, it ensures device operation beyond a standard five-year warranty period.

Abstract

Large language model (LLM) serving is increasingly dominated by agentic workloads, in which agents and their sub-agents accumulate context as KV cache across many requests, consuming substantial memory. High-bandwidth flash (HBF) is a promising solution, providing an order of magnitude greater capacity at HBM-class read bandwidth, but its finite write endurance is the key limiting factor. Our key insight is that KV cache should be placed across HBM and HBF by its lifetime. Placing shorter-lived data in HBM lets HBM absorb more of an agent run's writes and sends less of them to HBF. As the lifetime of KV cache in agentic serving is dictated by the harness, the program that orchestrates the agents, we analyze its behavior and identify three axes along which lifetime diverges, temporal, structural, and inter-worker. Guided by these observations, we present Lachesis, a lifetime-aware KV cache placement layer between the agent harness and the serving engine. At write time, it places each segment in HBM or HBF according to its lifetime, and frees its blocks once the segment is no longer read. In trace-driven simulation, Lachesis extends HBF lifetime by 1.19-3.13x over HBM-first placement, reaching 3.3-12.2 device-years. Even under continuous 24x7 operation at the full load a tight SLO admits, HBF outlasts its five-year warranty on the multi-agent trace.

Problem

• LLM serving
• KV cache
• agentic workloads
• high-bandwidth flash
• write endurance

Innovation

• KV cache placement
• High-bandwidth flash
• Lifetime-aware
• LLM agent serving
• Memory management