WorldScape Policy 2.0: Empowering Steerable World Action Modeling with Reasoning-Augmented Memory

2026-07-21

AI Summary

This work addresses the limitations of existing world action models, which struggle with task progress tracking and fine-grained multimodal alignment due to short temporal contexts, coarse-grained language supervision, and reliance solely on textual conditioning. To overcome these challenges, we propose a reasoning-enhanced, controllable world action model that integrates causal short-term visual memory with long- and short-term event memory. The model leverages semantic grounding and event-level multimodal alignment mechanisms to enable progress-aware autonomous planning under high-level instructions. Our approach combines DiT pre-filling, vision-language model (VLM)-based history organization, and autoregressive planning generation, and is pretrained on a newly curated multimodal dataset, ManipEvent-5M. Experiments demonstrate significant improvements in long-horizon planning, fine-grained instruction following, and contextual adaptability in both simulated and real-world environments.

Abstract

World Action Models (WAMs) offer a promising paradigm for robotic manipulation by jointly modeling visual state transitions and robot actions. However, existing WAMs are constrained by limited temporal context, coarse episode-level language supervision, and predominantly text-only conditioning, which hinder task-progress tracking and fine-grained language-video-action grounding while limiting visual-context reasoning and cross-embodiment transfer. In this paper, we introduce WorldScape Policy 2.0, a controllable WAM with reasoning-augmented long short-term memory. Its causal short-term visual memory supplies recent observations as DiT prefill to preserve local interaction dynamics, while its long short-term event memory organizes historical VLM outputs into global-history, local-active, and event-boundary representations for progress-aware retrieval. The retrieved history augments perception and autoregressively generated planning tokens, yielding an implicit subgoal condition for autonomous planning; semantic forcing further transfers event-level instruction semantics into this latent planning pathway. To establish fine-grained multimodal controllability, we construct ManipEvent-5M, an event-grounded embodied pretraining dataset containing nearly 5 million event segments with aligned action trajectories, episode-level task instructions, segment-level subtask captions, goal images, and video demonstrations. These designs provide a unified interface for autonomous planning from high-level instructions and controllable execution from fine-grained text, goal-image, or video-context prompts. Experiments in both simulation and real-world platforms demonstrate superior capabilities in long-horizon autonomous planning, fine-grained instruction following and in-context adaptation.

Problem

• World Action Models
• temporal context
• language-video-action grounding
• visual-context reasoning
• cross-embodiment transfer

Innovation

• World Action Models
• reasoning-augmented memory
• event-grounded pretraining
• multimodal controllability
• autonomous planning