Morphometric Imitation: From Morphology and Contact Aware Hand Retargeting to Sim-to-Real Visuomotor Policy

2026-09-23

AI Summary

This study addresses the challenges of morphological discrepancies between human and robotic hands, dynamic feasibility, and sim-to-real transfer in dexterous manipulation. To tackle these issues, this work proposes a three-stage framework that translates human interaction data into zero-shot visuomotor policies. First, a morphology metric optimization is designed to preserve contact patterns with high fidelity. Second, residual reinforcement learning is integrated to achieve dynamically feasible motion retargeting. Finally, visuomotor policy distillation enables seamless deployment from simulation to the real world. The proposed approach improves the contact F1 score by over 8 percentage points and achieves an 89.3% zero-shot manipulation success rate across 30 object categories in real-world settings.

Abstract

Human hand-object interactions (HOIs) provide a rich source of demonstrations for dexterous manipulation, but learning directly from them presents challenges in bridging morphology gaps, ensuring dynamical feasibility, and sim-to-real deployment. We present Morphometric Imitation, a three-stage framework that transforms reconstructed HOIs into zero-shot sim-to-real visuomotor policies. First, morphometric optimization (MMO) kinematically retargets human motion across hand morphologies while preserving demonstrated contacts. Second, residual reinforcement learning (RL) refines the kinematic reference using object pose and contact information from the human motion to produce dynamically feasible robot demonstrations. Third, these demonstrations are distilled into visuomotor policies. Across three robot hands and ten HOIs, MMO improves contact F1 over the strongest of five baselines by at least 8 points for every hand, while also improving the success rate of downstream dynamic retargeting by as much as 35 points. Ablations on the residual RL show complementary benefits from using object pose and contact information. Finally, the visuomotor policies achieve 89.3% zero-shot success in 300 real-world trials on 30 objects. Project page: $\href{https://morphometricimitation.github.io}{\text{this https URL}}$

Problem

• Hand-object interaction
• Morphology gap
• Dexterous manipulation
• Sim-to-real
• Visuomotor policy

Innovation

• Morphometric Optimization
• Hand Retargeting
• Residual Reinforcement Learning
• Visuomotor Policy
• Sim-to-Real