Robotic Manipulation by Imitating Generated Videos Without Physical Demonstrations

2025-07-01

AI Summary

This work addresses the challenge of teaching robots complex manipulation skills—such as pouring, wiping, and stirring—using only AI-generated videos, without physical demonstrations or robot-side training. Methodologically, it leverages text-to-video diffusion models to synthesize action videos, employs vision-language models to automatically filter semantically consistent video segments, extracts object motion trajectories via 6D pose estimation, and maps these trajectories to robot execution in an embodiment-agnostic manner. Its key contribution is the first demonstration of using purely synthetic video as supervision for end-to-end closed-loop transfer from generated visual data to real-world robot control. Experiments show that the approach achieves performance on par with human demonstrations, improves consistently with higher-generation video fidelity, and significantly outperforms baseline methods—including keypoint prediction and dense feature tracking—in both accuracy and generalization.

Technology Category

• Intelligent Robots: Manipulation
• Computer Vision: Diffusion Models for Vision
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

This work introduces Robots Imitating Generated Videos (RIGVid), a system that enables robots to perform complex manipulation tasks--such as pouring, wiping, and mixing--purely by imitating AI-generated videos, without requiring any physical demonstrations or robot-specific training. Given a language command and an initial scene image, a video diffusion model generates potential demonstration videos, and a vision-language model (VLM) automatically filters out results that do not follow the command. A 6D pose tracker then extracts object trajectories from the video, and the trajectories are retargeted to the robot in an embodiment-agnostic fashion. Through extensive real-world evaluations, we show that filtered generated videos are as effective as real demonstrations, and that performance improves with generation quality. We also show that relying on generated videos outperforms more compact alternatives such as keypoint prediction using VLMs, and that strong 6D pose tracking outperforms other ways to extract trajectories, such as dense feature point tracking. These findings suggest that videos produced by a state-of-the-art off-the-shelf model can offer an effective source of supervision for robotic manipulation.

Problem

• Enables robots to perform tasks without physical demonstrations
• Uses AI-generated videos for imitation learning in manipulation
• Filters and tracks object trajectories for robot retargeting

Innovation

• AI-generated videos replace physical demonstrations
• Vision-language model filters incorrect video outputs
• 6D pose tracker extracts object trajectories accurately