PatternDex: Learning Interaction Patterns to Guide Reinforcement Learning of Bimanual Dexterous Manipulation of Articulated Objects

2026-10-03

AI Summary

This study addresses the substantial embodiment gap and low success rates encountered when bimanual dexterous hands manipulate articulated objects. To overcome these challenges, we propose a policy training framework that learns interaction patterns from human demonstrations. Specifically, our method models hand-object kinematic correlations as reusable interaction primitives and generates robot-adapted motions to guide reinforcement learning policies. By integrating token sequence representations with contact point estimation, the framework enables efficient cross-embodiment transfer. Experimental results demonstrate that the proposed approach achieves an average success rate of 92.8% on the Allegro Hand, substantially outperforming the baseline at 52.2%. Furthermore, the learned policies successfully generalize to alternative robotic hands and are validated in real-world microwave-opening tasks.

Abstract

In this paper, we develop a method that enables bimanual dexterous hands to manipulate articulated objects with a high success rate without suffering from an embodiment gap. We observe that the correlation between hand motions and object motions is dictated by the object rather than the hands and can be learned from human-object demonstrations. Based on this observation, we propose PatternDex, a method that learns this correlation and represents it as a token sequence, which we call an interaction pattern. From this pattern, PatternDex estimates the wrist motions and contact points that fit the target robot, and then trains a reinforcement learning policy that exploits these estimates as guidance. Since the guidance fits the target embodiment, the policy explores only the actions that the target robot can execute and thus achieves high success rates. PatternDex also requires only simple fine-tuning to train a new robot, since it can reuse the learned interaction pattern. We evaluate PatternDex with bimanual dexterous hands on human demonstrations from the ARCTIC dataset. PatternDex achieves, on average, a 92.8% success rate with Allegro hands, while the state-of-the-art baseline achieves 52.2%. Also, it achieves success rates above 70% with three other robot hands after fine-tuning alone. Furthermore, we verify that the learned policy transfers well to a real-world task of opening a microwave. Videos and additional results are available at https://patterndex.github.io/PatternDex/

Problem

• bimanual dexterous manipulation
• articulated objects
• embodiment gap
• reinforcement learning
• human-robot transfer

Innovation

• Bimanual Dexterous Manipulation
• Interaction Pattern
• Reinforcement Learning
• Embodiment Gap
• Articulated Objects