Masked Visual Actions for Unified World Modeling

2026-07-21

AI Summary

This work addresses the challenge of aligning robotic actions with visual world models to jointly capture interaction priors and ensure physical controllability. It proposes “Masked Visual Actions,” a framework that represents actions as partially observable trajectories of arbitrary entities in video, establishing a unified control interface directly in pixel space. For the first time, this approach enables a single model to support both action-conditioned forward dynamics prediction and goal-directed inverse behavior synthesis. Relying only on limited real-world and simulated data, the method generalizes across diverse scenes and robotic platforms. By integrating video pretraining, masked trajectory inputs, and pixel-level action representations, the model generates imagined trajectories in downstream tasks that closely match actual executions, significantly enhancing performance in prediction, planning, and inverse action generation.

Abstract

Video models absorb rich priors over how the visual world moves, interacts, and responds to contact, making them promising substrates for robotic world modeling. The central challenge is how to communicate action to such models in a form aligned with the visual space in which they learned these interaction priors, yet still grounded in physical manipulation. We introduce Masked Visual Actions, a pixel-space control interface that expresses action as a partially revealed trajectory of an arbitrary entity in a video. Revealing robot motion makes the model act as a forward dynamics model that predicts the scene's response to low-level robot actions, while revealing desired object motion makes the same model recover robot behavior consistent with that outcome. Finetuned with only 15 hours of masked examples from real videos and simulation, a single checkpoint achieves strong visual fidelity and controllability across diverse scenes and multiple embodiments. In downstream manipulation settings, the model produces imagined rollouts whose outcomes correlate with real-world execution for policy evaluation, improves decision making by ranking candidate futures in model-based planning, and supports inverse modeling by synthesizing robot motion from desired object motion.

Problem

• world modeling
• visual actions
• robotic manipulation
• pixel-space control
• action representation

Innovation

• Masked Visual Actions
• pixel-space control
• unified world modeling
• forward and inverse dynamics
• video foundation models