Teleopit: A Full-Embodiment Humanoid Teleoperation System

2026-08-03

AI Summary

This work addresses the limitations of existing humanoid robot teleoperation systems, which often struggle to simultaneously achieve whole-body coordination, dexterous hand manipulation, and active viewpoint control while relying on specialized sensors or simplified commands. The authors propose a comprehensive VR-based embodied teleoperation framework that faithfully maps the operator’s body, hand, and head motions to the robot’s full body, configurable dexterous hands, and a 2-DoF active vision system. Key innovations include a wearable-free, optimization-based hand retargeting method—integrating normalized finger directions, fingertip closure, and thumb coordinate alignment—and a robustness-enhancing mechanism combining a history encoder with failure-aware retrospective sampling. Experiments on a physical robot demonstrate coordinated locomotion, manipulation, and viewpoint adjustment, with ACT and GR00T N1.7 policies trained on 96 demonstrations achieving task success rates of 90.0% and 95.0%, respectively.

Abstract

Humanoid teleoperation for demonstration collection requires coordinated whole-body motion, continuous dexterous hand control, and viewpoint control. Existing systems either simplify hand commands or depend on dedicated wearable sensors for fine-grained hand motion. We introduce Teleopit, a full-embodiment teleoperation system that maps body, hand, and head signals from VR to a humanoid body, configurable dexterous hands, and a 2-DoF active vision module. A history encoder and failure-aware rewind sampling improve the motion tracker on both motion-capture and live VR references. An optimization-based hand retargeter combines normalized finger directions, fingertip closure, and thumb-frame alignment to map human hand motion to different dexterous hands without tuning hand-specific objective or solver hyperparameters. Component experiments evaluate tracking success rate and retargeting behavior, while real-robot teleoperation demonstrates coordinated locomotion, manipulation, and viewpoint control. ACT and GR00T N1.7 policies trained on 96 successful demonstrations collected with Teleopit achieve task success rates of 90.0% and 95.0%, respectively, when deployed on the humanoid. The project page is available at https://botrunner64.github.io/teleopit-page.

Problem

• humanoid teleoperation
• dexterous hand control
• whole-body motion
• viewpoint control
• demonstration collection

Innovation

• full-embodiment teleoperation
• dexterous hand retargeting
• VR-based motion tracking
• failure-aware rewind sampling
• humanoid demonstration learning