Worth Remembering: Surprise-Gated Robot Episodic Memory

2026-06-02

AI Summary

General-purpose robots must efficiently store high-value experiences for unknown future tasks, yet retaining all experiences is infeasible. This work proposes a Bayesian surprise–based memory gating mechanism that unsupervisedly identifies and stores causally salient events within the semantically rich, deployment-agnostic latent space of V-JEPA-2, integrating them with a 4D scene graph spatial memory. To our knowledge, this is the first application of Bayesian surprise to robotic episodic memory selection, enabling unsupervised causal event segmentation. Experimental results demonstrate that the approach outperforms existing methods by at least 12% on robotic question-answering tasks involving temporal, spatial, and binary reasoning, and surpasses both supervised and non-causal baselines in event segmentation accuracy.

Abstract

Robots solving generalist tasks need to be able to ground instructions in their past experience, since humans may refer to notable past events when giving a task (e.g., ``Take me to where the chemical spill happened yesterday''). Since memory limits make storing all past events infeasible, long-term robot memory must be selective, ideally retaining only those episodes with high utility for future tasks. However, future tasks are not typically given a priori for generalist robots. To select generically useful memories, we propose Bayesian surprise as a gating mechanism for memory formation. We present an approach to compute surprise in a semantically rich deployment-agnostic latent space provided by V-JEPA-2. Using our gated episodic memory to augment 4D scene graph-based spatial memory, we show a consistent improvement over state-of-the-art benchmarks in robot question answering, outperforming prior robot memory methods by $\geq12\%$ for temporal, spatial, and binary questions, and surpassing the performance of supervised and non-causal methods with an unsupervised causal method in event segmentation tasks.

Problem

• robot memory
• episodic memory
• memory selection
• Bayesian surprise
• generalist robots

Innovation

• Bayesian surprise
• episodic memory
• V-JEPA-2
• 4D scene graph
• unsupervised causal learning