C2Dex: Contact-Consistent Reconstruction and Retargeting for Dexterous Manipulation from Monocular Video

2026-08-07

AI Summary

This work addresses the scarcity of high-quality robotic demonstration data by proposing a method to reconstruct physically plausible and stable hand–object interaction trajectories from monocular human manipulation videos and transfer them to dexterous hands. The approach aggregates frame-wise observations in the object’s canonical space to identify consistent contact points, which serve as explicit constraints to jointly guide hand reconstruction and cross-morphology retargeting. By integrating Laplacian-based interaction optimization with residual reinforcement learning, the method balances local geometric consistency with task success. Evaluated on DexYCB and TACO benchmarks, it achieves end-to-end success rates of 57.78% and 26.67%, respectively—substantially outperforming existing baselines—and demonstrates successful execution of diverse contact-intensive tasks on a real robot.

Abstract

High-quality demonstrations for dexterous robot manipulation are costly and difficult to collect, whereas monocular human videos provide a scalable source of diverse manipulation behaviors. However, transferring such demonstrations to dexterous robots remains challenging: monocular hand-object interaction (HOI) reconstruction often produces temporally unstable contacts and physically implausible interactions, while conventional retargeting methods struggle to preserve task-relevant contacts and local interaction geometry across different hand embodiments. We present C2Dex, a video-to-dexterous-manipulation framework built around a shared interaction representation: stable object-side contacts recovered by aggregating noisy frame-wise observations in the canonical object space. These stable contacts serve a dual role: as trajectory-level constraints that guide reconstruction toward temporally coherent and physically plausible human HOI trajectories, and as explicit transfer targets for the dexterous hand, where Laplacian interaction optimization preserves the local hand-object geometry across embodiments and residual reinforcement learning refines the trajectory in simulation. Experiments on DexYCB and TACO show that C2Dex achieves end-to-end trajectory success rates of 57.78% and 26.67%, respectively, substantially outperforming the strongest baselines (17.78% and 10.00%) under identical evaluation criteria. Real-robot replay experiments further demonstrate physical feasibility across diverse contact-rich manipulation tasks. Project page: https://k-jie.github.io/C2Dex/

Problem

• dexterous manipulation
• monocular video
• hand-object interaction
• contact consistency
• motion retargeting

Innovation

• contact-consistent reconstruction
• object-centric contact representation
• dexterous manipulation retargeting
• Laplacian interaction optimization
• monocular video to robot transfer