Cross-Model KV Cache Transfer in LLM Families: A Closed-Form Linear Mapping for Prefill Reuse

2026-08-04

AI Summary

This work addresses the computational inefficiency of repeated prefilling when switching between large language models of different scales. The authors discover, for the first time, a strong linear relationship between cross-model key-value (KV) caches and propose a training-free, closed-form ridge regression mapper to enable efficient cache reuse. Their method leverages RoPE removal, multi-layer selection, and few-shot calibration using only 500 FineWeb-Edu sequences, with an optional MLP-based nonlinear correction to enhance robustness. Evaluated across six model pairs, the linear mapping preserves 73%–98% of the original prefill accuracy while accelerating inference by 2.7–25×. For failure cases, the MLP correction improves HellaSwag retention by up to 37 percentage points, supporting seamless transitions across varying context lengths and multi-turn dialogues.

Abstract

Production deployments often swap between different-sized models in a family for cost-quality cascading, mid-conversation switching, and routing, and each swap forces the receiver to repay the prefill from scratch. We propose cross-model KV cache transfer, where the receiver reuses the source's KV cache, skipping prefill. We find that cross-model KV has substantial linear structure across matched-KV pairs, where source and target share KV head count and per-head dimension. On Qwen3 14B->32B, one source layer explains 56% of variance in the target's keys and 32% in values, rising to 79% and 65% with multiple source layers. Building on this, we design a closed-form ridge mapper that operates per head and proceeds in three steps. First, for each target layer we select the top-k most predictive source layers and concatenate their KV as input. Second, we strip RoPE from the keys before mapping, so the fit is position-free and reusable across context lengths. Third, we fit ridge regression on a small calibration set of 500 FineWeb-Edu sequences of 1,024 tokens each. Surprisingly, across six pairs in three families, this linear mapper retains 73-98% of the receiver's standalone-prefill accuracy on four pairs, while two degrade sharply. A nonlinear MLP recovers up to +37 pp HellaSwag retention on the failures. The mapper runs 2.7-25x faster than re-prefill and remains stable across multi-turn handoff, making cross-model KV cache transfer practical.

Problem

• KV cache transfer
• large language models
• prefill reuse
• model switching
• cross-model inference

Innovation

• KV cache transfer
• closed-form mapping
• linear structure
• prefill reuse
• cross-model inference