KAP: Bridging the Knowledge Selection-Runtime Consumption Gap in LLM Systems

2026-07-27

AI Summary

This work addresses the inefficiency in large language model inference caused by flattening structured knowledge selection signals into ordinary token sequences, which leads to redundant key-value (KV) cache accesses, increased latency, and reduced throughput. To overcome this limitation, the paper introduces Knowledge Access Planning (KAP), a novel paradigm that elevates structured knowledge selection to a first-class runtime primitive. KAP bridges knowledge selection and KV cache consumption through a unified intermediate representation and leverages the GraphSpec compiler-executor framework to enable plan-driven, efficient KV cache access. Experimental results demonstrate that, on long-context question answering tasks ranging from 4K to 128K tokens, KAP achieves comparable answer quality while accessing only 5.5% of the original KV states, substantially reducing latency and redefining the scaling efficiency of long-context generation.

Abstract

Modern LLM systems increasingly rely on knowledge-selection processes that produce high-value structured priors, such as ranked evidence, graph topology, multimodal alignment, and confidence signals. Yet LLM serving remains fundamentally oblivious to this rich structure: once such signals are serialized into a prompt, the backend observes only a flat token sequence, forcing dense and uniform consumption of the full key-value (KV) state during decoding. We term this architectural mismatch the Knowledge Selection-Runtime Consumption (KSRC) gap: richer contexts enlarge the full-prompt KV footprint and decode-time memory traffic, increasing latency and degrading throughput even when reasoning depends on only a small fraction of the context. To bridge the gap, we propose Knowledge Access Planning (KAP), a paradigm-shifting execution abstraction that elevates structured knowledge priors from passive prompt-construction hints into first-class physical execution artifacts. KAP establishes a universal intermediate representation (IR)-the runtime access plan-which compiles structured knowledge signals to govern physical KV access without altering logical prompt semantics, model weights, or training procedures. Through this IR, KAP shifts LLM serving from token-aware context consumption to plan-driven, knowledge-aware runtime consumption. We instantiate KAP with GraphSpec, a compiler-executor realization connecting structured knowledge selection to an LLM serving backend. We derive a phase-boundary model for the positive-speedup regime of plan-guided execution. Across 4K-128K long-context QA workloads, GraphSpec maintains answer quality comparable to full-context decoding while decoupling physical KV consumption from prompt length, reducing proposal-time KV access to 5.5% of source KV state at 128K, and fundamentally shifting the scaling trajectory of long-context generation.

Problem

• Knowledge Selection
• Runtime Consumption
• KV Cache
• Long-context LLM
• Architectural Gap

Innovation

• Knowledge Access Planning
• Runtime Access Plan
• KV Cache Optimization
• Structured Knowledge Priors
• Long-context LLM Serving