LUCID: Learning Embodiment-Agnostic Intent Models from Unstructured Human Videos for Scalable Dexterous Robot Skill Acquisition

2026-06-09

AI Summary

This work addresses the challenge of generalizing robot manipulation skills beyond expensive, embodiment-specific demonstration data. The authors propose a two-stage framework: first, a task-intent model is learned from unstructured internet videos; second, an embodiment-specific sensorimotor policy is trained in large-scale simulation, with both stages connected through a shared intent interface to enable zero-shot cross-embodiment transfer. This approach achieves, for the first time, zero-shot transfer of dexterous manipulation skills to multiple real robot embodiments using only in-the-wild videos as supervision. The method is validated across five tasks—including stirring, wiping, and sorting—and demonstrates that as little as one hour of smartphone-collected video suffices to successfully perform complex tasks such as T-block pushing and cable routing, substantially reducing data requirements while enhancing generalization.

Abstract

The most widely-adopted robot learning pipelines today learn skills from robot demonstrations or structured human data, which are expensive to collect and tied to specific embodiments. In contrast, unstructured human videos provide a scalable alternative. They contain diverse manipulation demonstrations across objects, scenes, and strategies, but are not directly connected to robot action. We propose LUCID, a two-stage framework that learns task intent from unstructured human videos drawn from internet-scale datasets and learns robot control in massively-parallel simulation. The intent model predicts short-horizon intent (what should happen next in the scene) from the current observation in closed loop. An embodiment-specific sensorimotor policy converts this intent into robot actions. The intent interface is shared across controllers, so the same intent model can be applied to different embodiments, from our primary dexterous hand to a parallel-jaw gripper. We evaluate LUCID on five real-world manipulation tasks: stirring, wiping, and binning supervised by only internet video, with zero-shot transfer to novel scenes and object instances; and push-T and cable routing supervised by 1 hr each of self-collected smartphone video. Project page: https://lucid-robot.github.io/.

Problem

• embodiment-agnostic
• unstructured human videos
• robot skill acquisition
• task intent
• scalable learning

Innovation

• embodiment-agnostic
• unstructured human videos
• intent modeling
• scalable robot learning
• zero-shot transfer