ConTrack: Constrained Hand Motion Tracking with Adaptive Trade-off Control

2026-06-02

AI Summary

This work addresses the challenge of transferring long-horizon, multi-contact human hand motions to robots for dexterous manipulation, a task hindered by the poor generalization and reliance on handcrafted reward functions in existing approaches. The authors propose a constrained reinforcement learning framework that formulates object trajectory tracking as a constraint rather than a reward objective. By integrating online dual-variable optimization, the method dynamically balances task accuracy and motion style during learning. Furthermore, an adaptive mid-episode reset buffer enables efficient training without per-sequence hyperparameter tuning. Experiments in both simulation and real-world robotic settings demonstrate significant improvements in task success rate and object pose accuracy, while faithfully reproducing human hand joint trajectories and contact timing with high fidelity.

Abstract

Human demonstrations provide strong priors for robot manipulation, yet it is non-trivial to transfer them to execute on real robots due to the kinematic gap. In dexterous manipulation, it remains challenging to track long-horizon, contact-rich sequences even in simulators: a reference-tracking policy must keep objects on their target trajectories while preserving demonstrated joint motion and contact timing. Existing approaches often rely on hand-crafted reward tuning that require per-sequence tuning and break under limited interaction budgets. We introduce ConTrack, a reinforcement learning (RL) framework that scales with tracking data. ConTrack treats object tracking as a constraint and allocates remaining control authority to motion fidelity, which allows it to adapt task--style trade-offs online using a dual-variable update. In addition, ConTrack also stabilizes long-horizon learning with an adaptive mid-trajectory reset library that reuses policy-reachable simulator states. Our qualitative and quantitative results in simulation tracking and real robot demonstrate that ConTrack improves success and object pose accuracy significantly over prior arts while preserving joint and contact fidelity. Website: https://www.lyt0112.com/projects/ConTrack.

Problem

• dexterous manipulation
• motion tracking
• kinematic gap
• contact-rich sequences
• human demonstration transfer

Innovation

• constrained reinforcement learning
• adaptive trade-off control
• dexterous manipulation
• motion tracking
• trajectory reset