Presentation, Not Mechanism: A Render Confound in Deprecation-Aware Memory Evaluation

2026-07-17

AI Summary

This study addresses the evaluation of evidence state revision in dynamic memory systems, highlighting that presentation format can confound assessments of underlying mechanisms. To disentangle memory mechanisms from interface effects, the authors propose a fixed rendering layout and evaluate flat retrieval, coarse-grained invalidation, and fine-grained RevisionLedger approaches on 2,907 high-consistency time-sensitive question-answer pairs sourced from platforms like GitHub and Wikipedia. The work introduces the concept of rendering confounding and mitigates it through render-matched control groups and a multi-annotator evaluation framework. Results reveal that fine-grained revision yields only marginal gains (+0.021 to +0.025), whereas coarse-grained invalidation significantly outperforms the baseline (+0.084) once presentation is controlled, demonstrating that query performance depends more critically on the adequacy of retained invalidated evidence than on mechanism complexity.

Abstract

AI systems increasingly retrieve from records that revise themselves: issue threads, encyclopedic histories, policy logs, and long conversations. The challenge is not only finding relevant evidence, but deciding which claims remain in force, which were superseded, and when to abstain. Structured memories promise to solve this with typed edges, temporal updates, and conflict status, yet evaluations often change mechanism and prompt presentation together. We study this as Evidence-State Revision, comparing flat retrieval, coarse edge invalidation, and fine-grained RevisionLedger on 2,907 high-agreement questions from GitHub, multi-repo issue histories, Wikipedia, and DyKnow-style temporal streams. A render-matched control (same layout, deprecation disabled) reveals the central confound: when a value is changed and later restored, RevisionLedger appears to beat a flat baseline by +0.182, but almost all the gain comes from easier presentation; the fine-grained mechanism residual is indistinguishable from zero (+0.021 to +0.025 across two judge families). After presentation is controlled, coarse invalidation is the only mechanism that pays for current-state queries, beating the fine ledger by 0.084; the same query-sufficiency principle says provenance mainly needs retained invalidated evidence, not richer typing. Memory evaluations should hold render fixed, and deprecation-aware systems should deploy the coarsest retained state that covers their queries.

Problem

• deprecation-aware memory
• render confound
• evidence-state revision
• memory evaluation
• temporal streams

Innovation

• render confound
• deprecation-aware memory
• Evidence-State Revision
• RevisionLedger
• coarse invalidation