DeVA: Decoupled Video-Action Model with physical guidance for robot policy learning

2026-07-27

AI Summary

Existing vision-language-action models struggle to capture physical dynamics and temporal causality due to their reliance on static pretraining objectives, necessitating large amounts of downstream demonstration data for effective policy learning. This work proposes a decoupled video-action dual-stream architecture that, for the first time, separates video prediction from action decision-making. The approach incorporates multi-level cross-modal feature transfer and introduces physics-aware supervision grounded in affordance and depth-based saliency. Evaluated on both simulated and real-world robotic tasks, the method achieves faster convergence with significantly less training data and substantially outperforms unified-architecture baselines, demonstrating that physically guided inductive biases effectively enhance policy generalization and sample efficiency.

Abstract

Generalizable robot manipulation requires policies that can anticipate how visual scenes evolve while executing language instructions. While recent Vision-Language-Action models benefit from large-scale pretraining, their predominantly static pretraining objectives provide limited supervision for physical dynamics and temporal causality, leaving control-relevant knowledge to be learned from downstream robot demonstrations. Video generative models offer a promising foundation by encoding rich spatiotemporal priors through future predictions. However, existing Video-Action Models either couple video and action prediction in a shared backbone, making policy adaptation harder to optimize, or under-utilize video information when guiding the action branch. In this work, we introduce DeVA, a Decoupled Video-Action model with specialized video and action experts, multi-level feature transfer, and physically salient guidance. DeVA transfers representations from multiple video layers to the action expert, enabling rich information exchange while making policy learning more tractable. It further supervises intermediate video features and the action stream with physically salient guidance (affordance/depth). Experiments on both simulation benchmarks and real-world deployment demonstrate strong performance with limited data, faster convergence than a unified architecture, and clear performance gains from physical guidance.

Problem

• robot policy learning
• physical dynamics
• temporal causality
• video-action models
• generalizable manipulation

Innovation

• Decoupled Video-Action Model
• Physical Guidance
• Multi-level Feature Transfer
• Affordance
• Vision-Language-Action