Geometric Retargeting: A Principled, Ultrafast Neural Hand Retargeting Algorithm

2025-03-10

AI Summary

This work addresses the real-time human-to-robot hand keypoint mapping problem in teleoperation. We propose GeoRT, an unsupervised, ultra-high-speed (1 kHz) neural retargeting algorithm. Unlike existing approaches relying on paired annotations or online optimization, GeoRT introduces a novel geometric objective function that jointly enforces motion fidelity, configuration-space coverage, response uniformity, pinch semantic consistency, and self-collision avoidance—significantly reducing hyperparameter sensitivity. The method integrates differential-geometric constraints, C-space-aware loss design, and an end-to-end regression architecture to achieve state-of-the-art accuracy. GeoRT has been deployed in a highly dexterous teleoperation system, enabling real-time motion correction for the DexGen controller.

Technology Category

• Intelligent Robots: Localization, Mapping, and Navigation
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Computer Vision: Remote Sensing / Geospatial AI

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments

Abstract

We introduce Geometric Retargeting (GeoRT), an ultrafast, and principled neural hand retargeting algorithm for teleoperation, developed as part of our recent Dexterity Gen (DexGen) system. GeoRT converts human finger keypoints to robot hand keypoints at 1KHz, achieving state-of-the-art speed and accuracy with significantly fewer hyperparameters. This high-speed capability enables flexible postprocessing, such as leveraging a foundational controller for action correction like DexGen. GeoRT is trained in an unsupervised manner, eliminating the need for manual annotation of hand pairs. The core of GeoRT lies in novel geometric objective functions that capture the essence of retargeting: preserving motion fidelity, ensuring configuration space (C-space) coverage, maintaining uniform response through high flatness, pinch correspondence and preventing self-collisions. This approach is free from intensive test-time optimization, offering a more scalable and practical solution for real-time hand retargeting.

Problem

• Develops ultrafast neural hand retargeting for teleoperation.
• Achieves high-speed, accurate conversion of human to robot hand keypoints.
• Eliminates manual annotation with unsupervised training and novel geometric functions.

Innovation

• Ultrafast neural hand retargeting algorithm
• Unsupervised training without manual annotation
• Novel geometric objective functions for retargeting