Now You Feel It, Now You See Me: Digital-Twin-based Teleoperation Interface for Dexterous Manipulation

2026-10-04

AI Summary

This study addresses the absence of force feedback and visual occlusion in bimanual robot teleoperation by proposing a digital twin-based augmented reality (AR) interaction framework. The system integrates bare-hand tracking and generates virtual viewpoints to effectively mitigate occlusion interference. Furthermore, it introduces a novel vision-haptic mapping mechanism that renders tactile data onto the robot's mesh model for intuitive force feedback, combined with occlusion-aware transparency rendering to enhance situational awareness. User studies demonstrate that the proposed framework significantly improves the quality of demonstration data acquisition in scenarios involving complex occlusions and fine-grained force control.

Abstract

Teleoperation is becoming increasingly important for collecting high-quality demonstrations to teach robots dexterous manipulation skills. For dexterous manipulation, bare-hand tracking provides a practical way to control robotic hands and demonstrate coordinated finger movements without gloves or exoskeletons. However, this type of teleoperation faces two key feedback limitations: a lack of force feedback and visual feedback of occluded region. The absence of force feedback hinders precise and safe manipulation, as operators must infer contact force visually rather than feel them directly. Occlusion by objects or other robot parts impairs the assessment of hand positions, approach distances, and pre-grasp configurations suited to the object's shape. To address these limitations, we propose a digital-twin-based augmented reality (AR) teleoperation interface that integrates bare-hand tracking, bimanual robotic hands, and virtual representations of the robot and task objects. The interface renders tactile measurements on robot hand meshes as visuo-force feedback. Furthermore, it offers a user-controlled virtual view panel or occlusion-aware transparency rendering to provide occlusion-mitigating visual feedback. We evaluated the interface through user studies involving Task 1 and Task 2, examining how force visualization and visual assistance support demonstration collection in tasks requiring careful force regulation and manipulation under occlusion.

Problem

• teleoperation
• dexterous manipulation
• force feedback
• visual occlusion
• bare-hand tracking

Innovation

• Digital Twin
• Teleoperation
• Dexterous Manipulation
• Augmented Reality
• Bare-hand Tracking