🤖 AI Summary
This work addresses the lack of a portable contract mechanism in existing systems for governing the reuse of resident key-value (KV) cache entries under memory pressure, which leads to ineffective arbitration when conflicts arise with active KV requests. The paper proposes a declarative mechanism for resident KV entries that binds reuse intent, predicates, lifecycle state, and feasibility outcomes into a runtime consistency contract, enabling traceable arbitration between active and resident KV entries. It introduces, for the first time, a telemetry-aware, declaration-level contract that transforms invisible resident KV evictions into reconstructible arbitration events, supporting scheduler-visible rejections with clear attribution. Prototype experiments based on vLLM demonstrate that the approach successfully handles 60 resident declarations alongside 70 active requests within an 80-block KV pool, achieving precise attribution and proactive rejection—outperforming conventional cache replacement policies.
📝 Abstract
KV-cache reuse mechanisms increasingly expose priority, duration, offload, routing hints, scheduler modes, and event streams. These mechanisms help preserve reusable prefixes, but they do not by themselves define a portable contract for accepted future-reuse state when resident KV and active live KV cannot both fit. We introduce resident KV claims, a conformance contract that binds future-reuse intent to a materialization predicate, lifecycle state, active/resident feasibility outcome, and claim-level telemetry. In controlled vLLM allocator probes, a 60-block resident claim and a 70-block active prefill exceed an 80-block usable KV pool. Write no-admit prevents the active request from becoming future reusable state, but it still allows active allocation to evict residents from the shared pool. A minimal vLLM prototype shows that hard protected resident claims convert this failure mode into scheduler-visible active refusal with direct blocking-claim attribution. The result is not a production speedup or a new cache-replacement algorithm. It is a runtime contract that turns unreported resident loss into reconstructable active/resident arbitration. A companion MicroRuntime and vLLM litmus suite distinguish ordinary eviction, soft priority, write no-admit, accepted hard claims, materialization failure, demotion, expiry, active refusal, and trace-level outcome reconstruction.