Learning What to Recall: Adaptive Multi-Cue Episodic Memory for World Models

2026-09-28

AI Summary

This study addresses the limitation of existing world models that rely on fixed retrieval criteria, hindering the accurate selection of effective long-term memories in dynamic environments. To overcome this, we propose the FAR framework, which trains a memory retriever using future-aware supervision. The core innovation lies in employing conditional log-likelihood as a predictive utility metric and integrating temporal, pose, visual, and audio multimodal cues to construct an adaptive scoring mechanism that remains future-blind at inference yet forward-looking during training, with memory recall implemented via diffusion models. Experimental results demonstrate that FAR significantly outperforms handcrafted methods across three distinct scenarios. By automatically adapting to reliable cues, the proposed approach substantially enhances memory retrieval accuracy in dynamic environments.

Abstract

World models predict future observations from current experience and actions, yet prediction can depend on observations seen far in the past. Episodic memory preserves past observations for later recall; however, as memory accumulates, it raises a fundamental question: which memories are useful for the current prediction, and which available retrieval cues should be trusted to find them? This is challenging because fixed criteria based on recency, pose overlap, or visual similarity can be unreliable across environments and queries. We propose Future-Aware Recall (FAR), a framework that learns episodic recall from future-aware predictive supervision and adaptive multi-cue scoring. During training, FAR measures predictive utility by the conditional log-likelihood of the realized future given recalled context, approximated by negative diffusion prediction loss, and uses it to train a retriever that remains future-blind at inference. The retriever learns cue-specific relevance and automatically determines which available retrieval cues, such as time, pose, vision, and audio, to trust for each query when selecting memories. Across three complementary settings, FAR outperforms hand-designed recall even with the same retrieval cues, automatically adapts which available cues to trust, and recalls the right history as the world changes. Together, these results establish FAR as a flexible, principled approach to episodic memory access in world models.

Problem

• World Models
• Episodic Memory
• Memory Retrieval
• Multi-Cue
• Prediction

Innovation

• World Models
• Episodic Memory
• Adaptive Multi-Cue Retrieval
• Future-Aware Supervision
• Diffusion Prediction Loss