An Internet for the KV Cache: Rethinking Classical Infrastructure Boundaries in the LLM Inference Age

2026-08-02

AI Summary

This work addresses the inefficiency in large language model (LLM) inference caused by redundant recomputation of key-value (KV) caches and the inability of current cloud infrastructures to support efficient cross-node cache reuse. To overcome these limitations, the authors propose an "Internet of KV Caches" architecture that treats the network as an active distribution channel, decoupling computation from storage. Inspired by content delivery networks, the system introduces an adaptive KV cache scheduling mechanism driven by model characteristics, infrastructure state, and application-level metrics. This approach dynamically balances bandwidth, latency, and cost to enable efficient cross-datacenter cache sharing, significantly reducing inference latency and operational expenses while enhancing context reuse efficiency in large-scale, heterogeneous LLM inference scenarios.

Abstract

LLM inference has become a global-scale, heterogeneous workload spanning agents, retrieval, tool-use, code execution and multi-modal reasoning. These workloads naturally enable context reuse from overlapping inputs, creating a major opportunity to store and reuse the contexts' KV Caches instead of recomputing them. However, model-side advances that shrink the KV Cache and system-side advances that reduce compute, storage, and transfer costs are evolve independently within legacy cloud boundaries. We argue that future inference infrastructure should allow decoupling of compute and KV Cache storage across cloud and datacenters. The network becomes an active distribution channel; bandwidth, latency and pricing directly determines how the KV Cache should be managed. We propose a vision for an Internet for the KV Cache, with KV Cache management working as a content-distribution system. In this view, KV Cache storage and recompute decisions are driven by model, infrastructure, and application metrics, to enable adaptive, content-driven decisions for minimizing latency and cost.

Problem

• KV Cache
• LLM inference
• infrastructure decoupling
• context reuse
• content distribution

Innovation

• KV Cache
• LLM inference
• content distribution
• compute-storage decoupling
• adaptive caching