Uncertainty-Aware Intention Prediction for Human-to-Robot Assembly Teleoperation

2026-06-06

AI Summary

This work addresses the challenge of scarce and costly robot demonstration data in human-robot collaborative teleoperated assembly by proposing an uncertainty-aware intention prediction framework. The method leverages human hand demonstrations as a scalable pretraining data source and integrates hierarchical transfer learning (MS-TCN++), conformal prediction for statistically reliable uncertainty quantification, and a vision-language model (VLM) to contextually refine low-confidence prediction segments. Evaluated with only 16 robot demonstrations, the approach improves the Edit score from 70.50 to 80.70. Following VLM-based correction, frame accuracy increases from 45.21% to 46.42%, while both F1@25 and F1@50 metrics show consistent gains, and the Edit score remains stable.

Abstract

In assisted teleoperation for human-robot collaboration, accurate intention prediction is critical for enabling timely and reliable robotic assistance during long-horizon manipulation and assembly tasks. These systems require continuous understanding of user behavior to recognize actions, anticipate intentions, and detect mistakes in real time. However, robot teleoperation demonstrations are costly and hardware-limited, whereas human demonstrations are easier to collect and provide rich temporal structure. To address this challenge, we propose an uncertainty-aware human-to-robot intention prediction framework that combines: (1) hierarchical transfer learning, where MS-TCN++ is pretrained on human hand demonstrations and fine-tuned on limited robot teleoperation data to capture low-level actions and high-level task intentions; (2) a conformal prediction module that provides frame-level prediction sets with statistical coverage guarantees for reliable uncertainty quantification and early intention estimation; and (3) VLM-guided segment correction, which selectively reviews low-confidence or temporally uncertain segments using visual and temporal context. The framework supports action recognition, temporal segmentation, intention anticipation, and mistake detection for assisted teleoperation. Experiments on robot assembly demonstrations with 22 action classes show that human-to-robot fine-tuning improves the robot test-set Edit score from 70.50 to 80.70 using only 16 robot demonstrations. Edit-safe VLM correction further improves frame accuracy from 45.21% to 46.42% and increases F1@25 and F1@50 while preserving the Edit score. These results show that human demonstrations provide scalable pretraining data for robust, uncertainty-aware robot action segmentation. Code and data: project website.

Problem

• intention prediction
• human-to-robot transfer
• uncertainty quantification
• teleoperation
• assembly tasks

Innovation

• uncertainty-aware prediction
• hierarchical transfer learning
• conformal prediction
• VLM-guided correction
• human-to-robot intention prediction