DexUMI: Using Human Hand as the Universal Manipulation Interface for Dexterous Manipulation

2025-05-28

AI Summary

This work addresses the challenge of generalizing dexterous manipulation skills across heterogeneous robotic hands. We propose a zero-shot transfer framework that leverages the human hand as a natural interface. Methodologically, we integrate a wearable haptic-feedback exoskeleton with kinematic self-adaptive mapping to achieve human–robot hand kinematic alignment; employ generative hand video inpainting to bridge visual domain gaps; and perform policy transfer via end-to-end imitation learning. Our key contribution is the first synergistic use of haptically enhanced data acquisition and high-fidelity visual replacement for cross-morphology skill transfer. We validate the framework on two structurally distinct dexterous robotic hands. Across multiple real-world manipulation tasks, it achieves an average success rate of 86%, demonstrating substantial improvements in both the generality and practical applicability of skill transfer.

Technology Category

Intelligent Robots: ManipulationComputer Vision: Generative Adversarial Networks (GANs) for VisionHumans and AI: Human-Aware Planning and Behavior Prediction

Application Category

Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAIResponsible Web: Machine-in-the-loop, human agency and autonomySearch and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

We present DexUMI - a data collection and policy learning framework that uses the human hand as the natural interface to transfer dexterous manipulation skills to various robot hands. DexUMI includes hardware and software adaptations to minimize the embodiment gap between the human hand and various robot hands. The hardware adaptation bridges the kinematics gap using a wearable hand exoskeleton. It allows direct haptic feedback in manipulation data collection and adapts human motion to feasible robot hand motion. The software adaptation bridges the visual gap by replacing the human hand in video data with high-fidelity robot hand inpainting. We demonstrate DexUMI's capabilities through comprehensive real-world experiments on two different dexterous robot hand hardware platforms, achieving an average task success rate of 86%.

Problem

• Bridging kinematics gap between human and robot hands
• Minimizing embodiment gap via hardware and software adaptations
• Transferring dexterous manipulation skills using human hand interface

Innovation

• Uses human hand as manipulation interface
• Bridges kinematics gap with wearable exoskeleton
• Replaces human hand with robot inpainting