Digital Twin-Driven VR Teleoperation with Multi-View Spatial Perception for Surgical Robots

2026-09-21

AI Summary

研究提出一种基于数字孪生的VR远程操作平台，通过多视角渲染和自然运动视差线索解决传统手术机器人单一视角限制和深度感知不足的问题。

Abstract

Current robot-assisted minimally-invasive surgery (RMIS) platforms provide a fixed console for the surgeon to view stereo endoscopic images and teleoperate instruments inside the patient. Several researchers have proposed the use of a head-mounted display (HMD) as a portable console, with video pass-through rendering of the endoscope images which, like the fixed console, restricts the operator to a single endoscopic viewpoint and limits depth perception. We present a digital twin-driven virtual reality (VR) teleoperation platform, where the digital twin is created from markerless perception of the surgical environment and streamed for display on the HMD. This overcomes the limitations of video pass-through by providing multi-view rendering and natural motion-parallax cues, enabling decoupling of the user's hand posture from strict instrument alignment. The system utilizes VR hand controllers to increase the teleoperation workspace and to improve the robustness and stability of instrument control compared to the hand tracking approach adopted by most prior systems. A 15-participant user study on the da Vinci Research Kit (dVRK) shows that our VR platform significantly outperforms a state-of-the-art HoloLens 2 mixed reality baseline, reducing path length by 86% and jerk by 95%, while achieving depth perception confidence comparable to or exceeding the traditional console across all conditions.

Problem

• Surgical Robots
• Depth Perception
• Multi-View Rendering
• Teleoperation

Innovation

• digital twin
• virtual reality (VR) teleoperation
• multi-view rendering
• natural motion-parallax cues
• VR hand controllers