SemPIC: Learning Semantic Position-Independent KV Caches

2026-07-30

AI Summary

Existing position-agnostic caching methods suffer from suboptimal performance in scenarios involving long-context reuse or shifts in the ordering of instructions, history, and documents due to their lack of future contextual awareness. To address this limitation, this work proposes SemPIC, a framework that employs behavior distillation to train a LoRA-enhanced Writer module to generate semantic-aware, hierarchical document key-value (KV) caches offline, while retaining the pretrained decoder as a fixed Reader. This design enables efficient cache reuse with full compatibility with standard KV interfaces. Furthermore, the approach innovatively incorporates KV gradient checkpointing to reduce peak training memory consumption without compromising end-to-end gradient flow. Experimental results across three models and four tasks demonstrate that SemPIC improves average micro-F1 from 0.53 (achieved by KV Packet) to 0.60, approaching the performance of Full Recompute at 0.62.

Abstract

Long-context retrieval and agentic workloads repeatedly reuse the same documents under changing instructions, histories, and document orders. Prefix caching cannot exploit this reuse, while position-independent caching (PIC) remains unreliable because independently compiled KV states lack the future context in which they will be consumed. Our diagnostics show that a learned boundary-conditioned baseline sharply reduces attention deviation near reusable-block boundaries but leaves interior and task-level residuals, motivating adaptation of the document representation itself. We present \emph{SemPIC}, which trains a LoRA-enabled Writer to compile native per-layer document KVs through behavioral distillation while retaining the pretrained decoder as an unchanged Reader. Adaptation is confined to offline cache construction, preserving the standard KV interface and cache-hit decoding path. We further introduce KV Gradient Checkpointing, which reduces peak training memory without severing gradients through cached KVs. Across three models and four tasks, SemPIC raises mean micro-F1 over KV Packet from 0.53 to 0.60, approaching Full Recompute at 0.62.

Problem

• long-context retrieval
• position-independent caching
• KV cache reuse
• document representation
• attention deviation

Innovation

• SemPIC
• position-independent caching
• behavioral distillation
• KV Gradient Checkpointing
• LoRA