$ ext{TREX}^2$: Dual-Reconstruction Framework for Teleoperated-Robot with EXtended Reality

2025-06-01

AI Summary

Existing XR teleoperation systems suffer from significant motion-to-motion (M2M) latency, resulting in large control errors and prolonged task completion times—primarily due to full reliance on unreliable network communication. This paper proposes a novel dual-state reconstruction framework that decouples control and rendering: real-time bidirectional state estimation is performed locally using on-device sensors to compensate for network latency and packet loss; we further introduce contention-aware GPU scheduling and bandwidth-adaptive point cloud scaling. As the first end-to-end open-source XR teleoperation framework, it achieves 69.8% and 73.1% reductions in control error over WLAN and cellular networks, respectively; across ten benchmark tasks, average error decreases by 57.2% and task completion time shortens by 37.7%, with only 6.7% runtime overhead.

Technology Category

• Intelligent Robots: State Estimation
• Planning, Routing, and Scheduling: Replanning and Plan Repair
• Computer Vision: Motion & Tracking

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Data management and stream processing for Web, mobile and wireless applications
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Robot teleoperation with extended reality (XR teleoperation) enables intuitive interaction by allowing remote robots to mimic user motions with real-time 3D feedback. However, existing systems face significant motion-to-motion (M2M) latency--the delay between the user's latest motion and the corresponding robot feedback--leading to high teleoperation error and mission completion time. This issue stems from the system's exclusive reliance on network communication, making it highly vulnerable to network degradation. To address these challenges, we introduce $ ext{TREX}^2$, the first end-to-end, fully open-sourced XR teleoperation framework that decouples robot control and XR visualization from network dependencies. $ ext{TREX}^2$ leverages local sensing data to reconstruct delayed or missing information of the counterpart, thereby significantly reducing network-induced issues. This approach allows both the XR and robot to run concurrently with network transmission while maintaining high robot planning accuracy. $ ext{TREX}^2$ also features contention-aware scheduling to mitigate GPU contention and bandwidth-adaptive point cloud scaling to cope with limited bandwidth. We implement $ ext{TREX}^2$ across three hardware settings, including simulated and physical robots, and evaluate it on 9,500 real-world teleoperation trials from the RoboSet dataset cite{kumar2024robohive}, covering single- and multi-step missions. Compared to state-of-the-art XR teleoperation frameworks, $ ext{TREX}^2$ reduces teleoperation error by up to 69.8% on WLAN and 73.1% on cellular networks with only 6.7% maximum runtime overhead. It also improves completion time by up to 47.7%, enabling smoother teleoperation. A real-world case study on ten stationary and mobile missions further shows $ ext{TREX}^2$ achieves up to 37.7% faster completion while lowering average teleoperation error by up to 57.2%.

Problem

• Reduces motion-to-motion latency in XR teleoperation
• Decouples robot control from network dependencies
• Improves teleoperation accuracy and completion time

Innovation

• Decouples robot control from network dependencies
• Uses local sensing to reconstruct delayed data
• Implements contention-aware scheduling and bandwidth adaptation