An Action Is Worth One Patch: Unified World-Action Modeling with PatchWAM

2026-09-22

AI Summary

研究通过PatchWAM模型统一视觉预测与动作生成，将连续动作视为一种兼容表示的补丁，无需专门的动作模块，实验表明该方法在多个基准测试中表现优异。

Abstract

Generative visual models offer a foundation for learning representations of physical dynamics, yet their extension to continuous control raises a fundamental question: do visual prediction and action generation require separate computational pathways? Existing approaches usually introduce trainable action heads or separate action experts to bridge low-dimensional states and high-dimensional visual representations. In this work, we explore whether the visual backbone's existing capacity can also support control when actions are expressed in a compatible representation. Thus, we introduce PatchWAM (Patch World-Action Model), which treats continuous actions as another type of patch through a fixed mapping called Action-as-Patch. This allows a single model to predict both how the robot should move and what the scene may look like afterward. Visual prediction and action generation become parts of the same generative process, without a dedicated action head or separate action expert. Experiments with subsampled training windows show gains over a matched dual-expert control, while benchmark evaluations reach 91.8% success rate on LIBERO-Plus and 96.12% on RoboTwin 2.0 in a full-data setting with additional augmented demonstrations. More broadly, the result suggests that capability need not be added where it can be inherited: the constraint on extending a generative backbone is the interface a new signal is written in, not the capacity to model it.

Problem

• visual prediction
• action generation
• computational pathways
• generative visual models
• continuous control

Innovation

• PatchWAM
• Action-as-Patch
• Unified Modeling
• Continuous Control
• Visual Prediction