Geometry-Aware Implicit Memory for Video World Models

2026-06-01

AI Summary

Existing video world models struggle to effectively retain information about scenes that have moved out of view during long-horizon rollouts, as both explicit and implicit memory approaches are often hindered by retrieval errors, redundant storage, or insufficient geometric modeling. This work proposes GIM-World, a novel framework that uniquely integrates explicit cross-view geometric constraints into an implicit memory architecture. It employs a lightweight Transformer encoder to compress historical observations into a fixed-size set of memory tokens and introduces a geometry head that queries camera poses to distill 3D scene structure from a frozen foundation model. An information-guided pruning strategy ensures memory compactness while preserving long-term geometric and visual consistency. Evaluated on the MIND dataset, GIM-World significantly outperforms current state-of-the-art explicit and implicit memory baselines.

Abstract

Video world models aim to simulate controllable visual environments, but long-horizon rollouts depend on what the model remembers after observations leave its native context window. Explicit memories retain frames or online 3D reconstructions, which can suffer from heuristic retrieval errors, redundant appearance storage, or reconstruction artifacts. Implicit memories compress history into a compact state, but existing designs are not explicitly constrained to encode cross-view scene geometry. We propose GIM-World, a geometry-aware implicit memory framework for video world models. A lightweight transformer encoder compresses variable-length history into fixed-size memory tokens, a camera-queryable geometry head distills 3D scene structure from a frozen foundation model into the memory during training, and an information-guided pruning rule keeps encoding cost bounded as history grows. The geometry teacher is discarded at inference, leaving a lightweight memory module. Experiments on MIND show that GIM-World better preserves long-horizon geometric and visual consistency than both explicit- and implicit-memory baselines.

Problem

• video world models
• implicit memory
• scene geometry
• long-horizon consistency
• memory compression

Innovation

• geometry-aware
• implicit memory
• video world models
• 3D scene structure
• memory compression