Problem
Research questions and friction points this paper is trying to address.
Egocentric human data
dexterous robot manipulation
in-the-wild demonstrations
real-world settings
visual challenges
EgoWild2Dex: Learning Dexterous Robotic Manipulation from In-the-Wild Human Experience
🤖 AI Summary
研究通过收集真实环境中的第一人称视角数据，使用EgoWild2Dex方法将人类操作经验转换为机器人可学习的信息，以提高双臂机器人的灵巧操作能力。
📝 Abstract
Egocentric human data provide a principled source of supervision for learning dexterous robot manipulation. Unlike prior approaches that often collect such data in constrained or specially constructed environments, we collect in-the-wild egocentric demonstrations in real-world settings, including homes, factories, and pharmacies, etc., where people perform their ordinary tasks while wearing head-mounted cameras. This collection protocol captures diverse workflows and hand-object interactions across long-tailed object and skill distributions, but also yields visually challenging observations due to scene clutter and head-motion-induced viewpoint changes (a mean cumulative rotation of $15.93^{\circ}$/s). To address these issues, we introduce EgoWild2Dex, which transfers in-the-wild ego-human experience to dual-arm robots with dexterous hands by jointly aligning unstable egocentric views and human motions with robot observations and actions, respectively. This work offers three benefits. First, we introduce GeoFormer, a differentiable geometric transformer that warps noisy human observations toward robot observations. Second, we design a human-robot training scheme to bridge the embodiment gap, enabling high task success with limited robot supervision. Third, we release EgoWild, a 538.9-hour in-the-wild egocentric human dataset comprising 179,049 episodes, 125,961 unique task descriptions, and 1,282 object categories. On real robots, EgoWild2Dex achieves an average success rate of 96.7% across three long-horizon bimanual dexterous manipulation tasks and an average object-level zero-shot success rate of 33.3%. The data, models, and code will be released.
Innovation
Methods, ideas, or system contributions that make the work stand out.
EgoWild2Dex
GeoFormer
in-the-wild egocentric data
dexterous robotic manipulation
🔎 Similar Papers
No similar papers found.
