Positions Are Not Facts: The Mismatch Between KV Caches and Memory

2026-09-27

AI Summary

This study addresses the issue that updating facts in large language models using KV cache as memory induces conflicts with prior records and degrades historical accuracy. By investigating the intrinsic mechanisms of fact updates within the KV cache through controlled multi-hop update experiments, learned probes, and textual detectors, this work systematically compares strategies such as masking and deletion-reconstruction regarding their effects on current and historical question answering. The findings reveal the critical role of preserving unchanged states and object dependencies, demonstrating that naive value replacement compromises historical accuracy in multi-hop reasoning. Experiments indicate that while masking reinforces new values, it is prone to unit-level errors; reconstructing old positions reduces historical accuracy by 20–41%, whereas transferring existing states incurs minimal impact.

Abstract

When a fact changes, how should a language model update the history stored in its key-value (KV) cache? Hiding the old record is cheap, but it may still contain needed details or answer questions about the past. We compare hiding whole records, hiding only replaced values, and deleting old text and recomputing the cache. In a controlled quantity task, masking makes all eight models prefer the new value more strongly, yet six lose complete answers through unit errors or failure to stop; keeping the unit preserves all current answers. Later states also retain useful information from earlier records: on multi-hop updates, rebuilding these states at unchanged positions lowers historical accuracy by 20-41 percentage points, whereas moving the existing states has little effect. Keeping object dependencies and unchanged revision passages prevents many losses. Recognition is a separate challenge. Learned readouts recover distinctions missed by fixed cache similarities on synthetic record pairs. On natural text, text-detector-selected masks show no clear advantage over random masks at the same rate in 14 same-model detector-generator comparisons. Query-dependent access can avoid some losses, with additional storage or access costs. These findings identify what must be preserved beyond the replaced value when using a KV cache as updatable memory.

Problem

• KV cache
• language model
• memory update
• fact editing
• information preservation

Innovation

• KV cache
• memory update
• fact editing
• multi-hop reasoning
• learned readout