Echo-Memory: A Controlled Study of Memory in Action World Models

2026-06-08

AI Summary

Existing action-conditioned world models often suffer from silent scene changes after the camera departs due to memory failure, and lack a standardized benchmark for fairly comparing memory mechanisms. This work introduces a controlled experimental framework that systematically disentangles four key dimensions of memory—capacity, compression, retrieval, and recurrence—while fixing the video diffusion backbone, optimizer, action representation, and evaluation protocol. A three-branch evaluation scheme reveals that replay fidelity alone is insufficient to assess true memory capability. By unifying the action-to-video interface to compare mechanisms including raw context, compressed memory, spatial summaries, and state-space recurrence, the study finds that raw context significantly improves open-domain return performance, while block-wise state-space recurrence achieves the best results on this task, whereas compact compression tends to discard critical information.

Abstract

We present \textbf{Echo-Memory}, a controlled study of memory mechanisms in action-conditioned world models. These models generate multi-segment videos from a first frame, text prompt, and camera-action sequence, but their central failure is often memory rather than local image synthesis: after the camera leaves and returns, the scene or salient object may silently change. Existing memory designs are hard to compare because gains are entangled with backbone, training, retrieval, and evaluation differences. Echo-Memory fixes the action-to-video interface and varies only how history is stored and read by the generator. Under a shared video diffusion backbone, optimizer, camera-action representation, sampler, and evaluation pipeline, we compare raw context, compression-based memory, spatial summaries with different read-out paths, and state-space recurrence. This matched matrix separates four otherwise conflated axes: \emph{capacity}, \emph{compression}, \emph{read-out}, and \emph{recurrence}. We also evaluate memory through a three-branch protocol: replay quality, in-domain loop revisit, and open-domain return probes. The branches routinely disagree, showing that replay fidelity is not a sufficient proxy for remembering a world. Three findings follow. Raw context is a strong capacity baseline and improves open-domain return far more than it improves replay metrics. Compactness is not a free substitute for capacity: aggressive spatial and hybrid-compression memories lose the salient evidence needed for return. Finally, block-wise state-space recurrence is the strongest open-domain return mechanism in our matrix, showing that the structure of implicit memory matters as much as the decision to use it. These results provide a compact protocol for studying memory in action world models beyond isolated replay metrics.

Problem

• memory
• world models
• action-conditioned video generation
• scene consistency
• camera revisit

Innovation

• memory mechanisms
• action-conditioned world models
• state-space recurrence
• controlled ablation study
• open-domain return