Learning Diverse Humanoid Tasks via Synthetic Video Scenarios without Real World Data

2026-07-22

AI Summary

This work addresses the limitations in humanoid robot skill learning imposed by the high cost of acquiring real human demonstration data, insufficient motion diversity, and inadequate coverage of individual variability. The authors propose a purely synthetic learning framework that eliminates the need for real human demonstrations by leveraging generative AI to translate textual prompts into diverse and realistic human motion sequences. These synthesized motions are embedded into virtual video scenes to train robots within a simulation environment through a combination of reinforcement and imitation learning. This approach achieves, for the first time, fully synthetic-data-driven acquisition of diverse motor skills. Evaluated across four simulated tasks, the method not only successfully accomplishes target objectives but also demonstrates strong generalization capabilities to complex motion variations, thereby overcoming reliance on real-world demonstration data.

Abstract

The human-like morphology of humanoid robots grants them exceptional potential for agile and versatile motor capabilities, but it also introduces significant challenges in acquiring complex skills. Traditional Learning-from-Demonstrations methods are often constrained by the high cost of collecting real-world data, the difficulty of capturing motion-specific behaviors, and the limited diversity of demonstrations across individuals. Moreover, even for the same task, humans may execute the motion in multiple distinct ways. In this paper, we propose a new framework that leverages the power of Generative AI to convert textual prompts into realistic and diverse sequences of human body movements, enabling the robot to observe multiple variations of how a single task can be performed. These synthetic demonstrations are then used as a training resource, allowing the robot to learn a broad range of task-execution styles without requiring direct human intervention. We evaluate the proposed method across four simulation scenarios. Experimental results show that the robot not only completes the tasks successfully but also demonstrates strong adaptability to complex variations in motion.

Problem

• humanoid robots
• learning from demonstrations
• motion diversity
• synthetic data
• motor skills

Innovation

• Generative AI
• synthetic demonstrations
• humanoid robots
• text-to-motion
• learning from demonstration