UniTacHand: Unified Spatio-Tactile Representation for Human to Robotic Hand Skill Transfer

2025-12-24

AI Summary

Heterogeneity between human and robotic tactile signals impedes policy transfer. Method: This paper proposes a unified spatiotemporal tactile representation framework for zero-shot dexterous hand policy transfer. Its core innovations are: (1) a novel 2D morphologically consistent tactile projection space built upon the MANO hand model, enabling geometric alignment between human hands (equipped with tactile gloves) and robotic hands (embedded with sensor arrays); (2) cross-domain self-supervised contrastive learning for representation alignment using only 10 minutes of paired data; and (3) support for zero-shot policy transfer and human-robot mixed collaborative training. Results: Experiments demonstrate successful zero-shot tactile manipulation of unseen objects on real robots, significantly reducing reliance on real-robot tactile data while improving data efficiency and generalization across objects and tasks.

Technology Category

• Intelligent Robots: Manipulation
• Machine Learning: Learning with Manifolds
• Computer Vision: Representation Learning for Vision

Application Category

• Economics, Online Markets and Human Computation: The sharing economy
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Tactile sensing is crucial for robotic hands to achieve human-level dexterous manipulation, especially in scenarios with visual occlusion. However, its application is often hindered by the difficulty of collecting large-scale real-world robotic tactile data. In this study, we propose to collect low-cost human manipulation data using haptic gloves for tactile-based robotic policy learning. The misalignment between human and robotic tactile data makes it challenging to transfer policies learned from human data to robots. To bridge this gap, we propose UniTacHand, a unified representation to align robotic tactile information captured by dexterous hands with human hand touch obtained from gloves. First, we project tactile signals from both human hands and robotic hands onto a morphologically consistent 2D surface space of the MANO hand model. This unification standardizes the heterogeneous data structures and inherently embeds the tactile signals with spatial context. Then, we introduce a contrastive learning method to align them into a unified latent space, trained on only 10 minutes of paired data from our data collection system. Our approach enables zero-shot tactile-based policy transfer from humans to a real robot, generalizing to objects unseen in the pre-training data. We also demonstrate that co-training on mixed data, including both human and robotic demonstrations via UniTacHand, yields better performance and data efficiency compared with using only robotic data. UniTacHand paves a path toward general, scalable, and data-efficient learning for tactile-based dexterous hands.

Problem

• Aligns human and robotic tactile data for skill transfer.
• Enables zero-shot policy transfer to unseen objects.
• Improves learning efficiency with mixed human-robot data.

Innovation

• Aligns human and robotic tactile data via unified representation
• Projects tactile signals onto consistent 2D surface space
• Uses contrastive learning with minimal paired data for alignment