DyDexHandover: Human-like Bimanual Dynamic Dexterous Handover using RGB-only Perception

2025-09-22

AI Summary

Dual-arm aerial object handover poses challenges including weak visual perception, difficult inter-agent coordination, and unnatural motion generation. This paper proposes the first end-to-end multi-agent reinforcement learning framework operating solely on raw RGB images—requiring no dynamics models, depth sensors, or strong domain priors. Our method integrates a vision-based policy network with a human motion regularization mechanism, enabling natural, human-like collaboration in a dual-arm simulation environment built in Isaac Sim. Experiments show near-perfect handover success (99%) on trained objects and 75% cross-object generalization—substantially outperforming baselines. Generated throw-and-catch trajectories exhibit high kinematic plausibility and behavioral naturalness. Key contributions include: (1) the first purely RGB-driven dual-arm aerial handover system; (2) a transferable human-policy regularization paradigm that jointly optimizes task performance, generalization, and biomechanical plausibility.

Technology Category

• Intelligent Robots: Manipulation
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Agentic search
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Dynamic in air handover is a fundamental challenge for dual-arm robots, requiring accurate perception, precise coordination, and natural motion. Prior methods often rely on dynamics models, strong priors, or depth sensing, limiting generalization and naturalness. We present DyDexHandover, a novel framework that employs multi-agent reinforcement learning to train an end to end RGB based policy for bimanual object throwing and catching. To achieve more human-like behavior, the throwing policy is guided by a human policy regularization scheme, encouraging fluid and natural motion, and enhancing the generalization capability of the policy. A dual arm simulation environment was built in Isaac Sim for experimental evaluation. DyDexHandover achieves nearly 99 percent success on training objects and 75 percent on unseen objects, while generating human-like throwing and catching behaviors. To our knowledge, it is the first method to realize dual-arm in-air handover using only raw RGB perception.

Problem

• Enabling dual-arm robots to perform dynamic in-air object handovers
• Overcoming limitations of depth sensors and strong priors in handover tasks
• Achieving human-like bimanual coordination using only RGB perception

Innovation

• RGB-only perception for bimanual handover
• Multi-agent reinforcement learning for end-to-end policy
• Human policy regularization for natural motion