Knowing the Store: What a Memory Backend Must Write Down Before an Agent Can Read It

2026-10-03

AI Summary

This study addresses the vulnerability of agent memory systems to missing or erroneous backend metadata, which frequently leads to the retrieval of outdated records and constrains agents’ metacognitive monitoring capabilities. To investigate this issue, we construct synthetic memory repositories and conduct controlled metadata experiments using an independently scored metacognitive judgment paradigm. We systematically compare the performance of large language models against cosine similarity retrieval across varying candidate list lengths. Our findings reveal that current backend metadata deficiencies severely mislead all reader architectures. Furthermore, while LLMs exhibit no significant advantage over baselines with short candidate lists, stronger models outperform baselines exclusively in long-list scenarios. Ultimately, this work empirically validates the critical role of metadata quality in ensuring the reliability of agent memory systems.

Abstract

An agent with long-term memory can answer from a record it should no longer use, such as a plan the user later cancelled. We ask what a memory store must expose for an agent to know this before retrieving anything, and we score that judgment on its own, as metamemory monitoring. Readers see only a value-free summary: record counts by lifecycle state and a list of attribute names. Each of 148 questions is asked against three versions of one store that differ in one attribute, so wording cannot give the answer away. What a model adds depends on the length of that list. On the short lists the benchmark's ground truth produces (2.6 names on average), none of five language models reliably beats a cosine-similarity lookup over the names, and the three stronger ones are equivalent to it within 0.05. Under a control that fixes counts and list length, none is reliably above it and none is shown equivalent. On lists longer than the stores we tested write, padded to 60 names, the lookup loses 0.18; GPT-5.6 Luna and Sol lose 0.06 to 0.07 and lead it by 0.13 to 0.14, while the other three fall with it. The lead holds for Sol against a sibling padded to the same length and counts. Reworded to share no word with the names, the lookup loses 0.02 and one stronger reader edges 0.04 ahead. On the short lists the three stronger readers lead only where the summary counts a state without naming it, and one count feature closes that lead within a question, though not when items are pooled. The backends we tested omit or misstate this information: under FR-Bank's own metadata, GPT-4.1 mini, Haiku 4.5 and the lookup fall from 0.73, 0.82 and 0.75 to 0.60, 0.71 and 0.59. In these tests, what the store wrote down limited every reader we gave it to. All stores are synthetic; control, whether a better judgment yields a better answer, is left to a later study.

Problem

• long-term memory
• metamemory monitoring
• memory backend
• AI agents
• record lifecycle

Innovation

• Metamemory Monitoring
• Memory Backend
• Long-term Memory Agents
• Controlled Benchmark
• Metadata Design