Beyond the Shadows of Plato's Cave: Evaluating False Memory in Autonomous Agents via Counterfactual Reasoning

2026-09-30

AI Summary

Autonomous agents are susceptible to false memories arising from environmental shifts or spurious correlations, which remain difficult to distinguish from standard generalization failures. This work proposes FAME, a framework that traces agent belief evolution through counterfactual reasoning and estimates potential concept drift via latent state modeling to quantitatively evaluate false memories. Notably, this approach requires no additional training, reward engineering, or answer sampling, enabling effective decoupling of false memories from ordinary generalization failures. Experimental results demonstrate that FAME achieves AUROC scores ranging from 76.2% to 96.7% across mathematical, code, and complex reasoning benchmarks, outperforming the strongest baseline by margins of 3.4% to 23.3%.

Abstract

Autonomous agents increasingly rely on memory to generalize beyond their training environments. However, agents are bounded by what they have seen and believed, and leveraging such memories in unseen environments can introduce biases into their internal beliefs. We formalize this phenomenon as \textit{false memory}, which can arise from spurious correlations, environment shifts, and knowledge conflicts. Despite its importance, false memory is difficult to evaluate because it stems from agent internal beliefs and is easily confounded with ordinary generalization failures. Therefore, we propose FAME, a training-free framework that evaluates false memory through the evolution of agent beliefs under counterfactual reasoning. Specifically, counterfactual scenarios reveal how beliefs change as the latent concept of memory shifts under hypothetical interventions; thus, measuring the resulting concept drift provides a signal for distinguishing faithful versus false memory. Such concepts can be estimated from agent hidden states before answer generation, avoiding the need for reward design or answer sampling. Empirical experiments reveal that simply monitoring answers often fails to detect false memory, while FAME achieves AUROCs of 76.2% - 96.7% across false-memory settings, and outperforms the best baseline by 3.4% - 23.3% across realistic benchmarks, spanning math reasoning (GSM-Symbolic), code generation (GitChameleon), and complex reasoning (BigBench-Hard). We further release corresponding counterfactual templates and facilitate future research on false memory.

Problem

• false memory
• autonomous agents
• evaluation
• internal beliefs
• counterfactual reasoning

Innovation

• False Memory
• Counterfactual Reasoning
• Training-free Framework
• Concept Drift
• Autonomous Agents