Real-time Monocular 2D and 3D Perception of Endoluminal Scenes for Controlling Flexible Robotic Endoscopic Instruments

2026-02-16

AI Summary

This study addresses the challenges of precise localization and tissue-distance perception for flexible instruments in endoluminal surgery, which critically limit procedural efficiency and safety. To overcome these limitations, we propose a monocular vision-based perception framework that integrates high-fidelity physics simulation with deep learning, enabling, for the first time, real-time joint 2D/3D pose estimation and tissue-distance measurement of continuum surgical instruments within complex endoluminal environments. The developed simulator accurately models instrument dynamics, facilitating robust training and deployment of the perception algorithm. Experimental results demonstrate that the proposed method reduces task completion time by over 70% in trajectory-following tasks while significantly enhancing both manipulation accuracy and contextual scene understanding.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Computer Vision: Low Level & Physics-based Vision
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Endoluminal surgery offers a minimally invasive option for early-stage gastrointestinal and urinary tract cancers but is limited by surgical tools and a steep learning curve. Robotic systems, particularly continuum robots, provide flexible instruments that enable precise tissue resection, potentially improving outcomes. This paper presents a visual perception platform for a continuum robotic system in endoluminal surgery. Our goal is to utilize monocular endoscopic image-based perception algorithms to identify position and orientation of flexible instruments and measure their distances from tissues. We introduce 2D and 3D learning-based perception algorithms and develop a physically-realistic simulator that models flexible instruments dynamics. This simulator generates realistic endoluminal scenes, enabling control of flexible robots and substantial data collection. Using a continuum robot prototype, we conducted module and system-level evaluations. Results show that our algorithms improve control of flexible instruments, reducing manipulation time by over 70% for trajectory-following tasks and enhancing understanding of surgical scenarios, leading to robust endoluminal surgeries.

Problem

• endoluminal surgery
• monocular perception
• flexible robotic instruments
• real-time 2D/3D perception
• instrument-tissue distance

Innovation

• monocular perception
• continuum robot
• endoluminal surgery
• physically-realistic simulation
• learning-based 3D reconstruction