π€ AI Summary
This work addresses the challenge of knowledge evolution in traditional retrieval-augmented generation (RAG), which often retrieves outdated facts. The authors propose MemStrata, a temporally aware retrieval memory system that introduces, for the first time, a dual-temporal ledger mechanism. This mechanism automatically deprecates superseded facts by applying deterministic replacement rules to (subject, predicate, object) triples, eliminating the need for similarity thresholds or large language model intervention. MemStrata maintains strong static knowledge recall while significantly improving temporal accuracy. Experimental results show that on six local benchmarks, MemStrata matches RAG in static performance, achieves 0.95β1.00 accuracy on evolving knowledge (compared to RAGβs 0.20β0.47), reduces outdated errors to nearly 0%, and incurs only 2.1 seconds of retrieval latency.
π Abstract
Retrieval-augmented generation (RAG) gives agents access to accumulated knowledge, but has no model of time. When a fact changes (e.g., a function is renamed or API restructured), RAG retrieves both the stale and current value with near-identical embedding similarity. The agent then either abstains or serves the superseded fact. We show this is a structural problem: on a calibrated dataset, cosine similarity distinguishes a contradicted fact from a duplicated one with AUROC 0.59 (near chance), as contradictions are often more embedding-similar to the original than rephrased duplicates.
We present MemStrata, a retrieval memory maintaining temporal validity. It stores facts like RAG, preserving static recall, but when a fact's value is contradicted, a deterministic (subject, relation, object) supersession rule retires the stale value in a bi-temporal ledger - with no similarity threshold and no LLM call. Across six benchmarks run locally with a 7B model, MemStrata ties RAG on static knowledge and reaches 0.95-1.00 accuracy on evolving knowledge (where RAG reaches 0.20-0.47). The central result is the stale-fact-error rate: when required to answer, RAG serves superseded values 15-40% of the time; MemStrata drives this to ~0%, a failure class RAG cannot avoid. MemStrata achieves this at retrieval latency (~2.1s) versus ~16-18s for LLM-reranking baselines. We release the harness, datasets, and a marker-free evaluation protocol for memory under knowledge evolution.