InferScale: GPU-Native KV Injection for Personalized LLM Serving

2026-07-29

AI Summary

This work addresses the significant increase in Time-To-First-Token (TTFT) caused by repeatedly injecting user memory into prompts in personalized large language model services, where TTFT grows with the amount of retrieved content. To mitigate this, the authors propose an efficient memory injection mechanism that precomputes and caches key-value (KV) states of memory segments on the GPU, enabling direct injection into vLLM’s paged cache during inference and eliminating redundant prefill computations. The approach innovatively introduces Chunked RoPE to support positional encoding under dynamic memory concatenation and employs Context-Window Encoding to preserve cross-memory contextual coherence—all without modifying the inference engine or fine-tuning the model. Experiments on the LoCoMo dataset (k=50) demonstrate a 72–79% reduction in TTFT (3.6–4.8× speedup), a 3.7–4.5× throughput improvement, and only a minor drop in accuracy to 60.3%.

Abstract

Large language models are increasingly deployed with persistent personalized context, such as accumulated memory profiles or long conversation histories, that is shared across a user's many requests. Production memory systems (e.g., Mem0, MemGPT, and Zep) retrieve a relevant subset of this memory and inject it into the prompt, forcing the serving engine to repeatedly prefill the same content. As the retrieval budget grows, time-to-first-token (TTFT) increases even though the underlying memory is reused across requests. We present InferScale, a GPU-native LLM memory system that replaces repeated prompt prefilling with reusable KV state. InferScale precomputes each memory fact's KV representation, stores it alongside a semantic embedding on the GPU, retrieves relevant facts at serving time, and injects their KV directly into vLLM's paged cache. To support dynamically assembled memories under rotary position embeddings, we introduce Chunked RoPE, which stores keys before rotation and applies their serving-time positions during injection. However, encoding memory facts independently omits the cross-fact context available during joint prefilling. We mitigate this with Context-Window Encoding, which encodes each memory fact together with a small window of preceding conversation context while caching only the target fact's KV. InferScale is implemented through vLLM's KV-connector interface, requiring neither engine modifications nor model fine-tuning. Across three open-weight models on LoCoMo, InferScale keeps TTFT nearly constant as the retrieval budget increases: at k=50 it reduces TTFT by 72-79% (3.6-4.8x), achieves 60.3% accuracy versus 63.3% for Mem0 without serving-time recomputation, and delivers 3.7-4.5x the throughput under concurrent load. Reusable KV state thus decouples memory-conditioned serving latency from retrieved-context size while preserving application quality.

Problem

• personalized LLM serving
• KV cache
• time-to-first-token
• memory injection
• prompt prefilling

Innovation

• reusable KV cache
• Chunked RoPE
• Context-Window Encoding
• GPU-native memory system
• personalized LLM serving