RobustDexGrasp: Robust Dexterous Grasping of General Objects from Single-view Perception

2025-04-07

AI Summary

To address the limited robustness and generalization of dexterous grasping under single-view perception—particularly for unknown objects, arbitrary poses, and external disturbances—this paper introduces a hand-centered local shape representation coupled with a hybrid curriculum learning framework. It achieves zero-shot dynamic dexterous grasping without expert demonstrations or full-observation supervision for the first time. Our approach unifies vision-tactile privileged information distillation, imitation learning, and reinforcement learning, while incorporating dynamic randomization and explicit observation noise modeling to enhance disturbance resilience. In simulation, the method achieves a 97.0% success rate across 247,786 diverse objects; on physical hardware, it attains 94.6% success on 512 previously unseen objects—significantly outperforming existing baselines. Moreover, it demonstrates strong adaptability to motion occlusion and external force perturbations.

Technology Category

• Intelligent Robots: Manipulation
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Computer Vision: Representation Learning for Vision

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

Robust grasping of various objects from single-view perception is fundamental for dexterous robots. Previous works often rely on fully observable objects, expert demonstrations, or static grasping poses, which restrict their generalization ability and adaptability to external disturbances. In this paper, we present a reinforcement-learning-based framework that enables zero-shot dynamic dexterous grasping of a wide range of unseen objects from single-view perception, while performing adaptive motions to external disturbances. We utilize a hand-centric object representation for shape feature extraction that emphasizes interaction-relevant local shapes, enhancing robustness to shape variance and uncertainty. To enable effective hand adaptation to disturbances with limited observations, we propose a mixed curriculum learning strategy, which first utilizes imitation learning to distill a policy trained with privileged real-time visual-tactile feedback, and gradually transfers to reinforcement learning to learn adaptive motions under disturbances caused by observation noises and dynamic randomization. Our experiments demonstrate strong generalization in grasping unseen objects with random poses, achieving success rates of 97.0% across 247,786 simulated objects and 94.6% across 512 real objects. We also demonstrate the robustness of our method to various disturbances, including unobserved object movement and external forces, through both quantitative and qualitative evaluations. Project Page: https://zdchan.github.io/Robust_DexGrasp/

Problem

• Enabling zero-shot dynamic dexterous grasping of unseen objects
• Enhancing robustness to shape variance and uncertainty
• Adapting to disturbances with limited observations

Innovation

• Reinforcement-learning-based zero-shot dynamic grasping
• Hand-centric object representation for robustness
• Mixed curriculum learning with imitation and RL