EasyVis2: A Real Time Multi-view 3D Visualization System for Laparoscopic Surgery Training Enhanced by a Deep Neural Network YOLOv8-Pose

2024-12-21

AI Summary

To address the low accuracy and high computational latency of real-time multi-view 3D visualization in laparoscopic surgical training, this paper proposes a gesture-free real-time 3D visualization system. The system integrates a miniature array endoscopic camera trocar to construct a dedicated multi-view endoscopic image dataset; it pioneers the adaptation of lightweight YOLOv8-Pose to endoscopic environments, combined with a cross-view pose fusion strategy and a real-time surface modeling–scene overlay rendering pipeline. Compared to baseline methods under identical camera configurations, the system reduces 3D reconstruction error by 32%, decreases per-frame processing time by 41%, and achieves a key-instrument landmark detection mAP of 98.7%. Key contributions include: (1) the first lightweight pose estimation adaptation framework tailored for endoscopic vision; (2) a geometric–learning collaborative reconstruction paradigm leveraging cross-view constraints; and (3) an end-to-end real-time 3D surgical scene visualization system.

Technology Category

• Computer Vision: 3D Computer Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Multi-instance/Multi-view Learning

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

EasyVis2 is a system designed to provide hands-free, real-time 3D visualization for laparoscopic surgery. It incorporates a surgical trocar equipped with an array of micro-cameras, which can be inserted into the body cavity to offer an enhanced field of view and a 3D perspective of the surgical procedure. A specialized deep neural network algorithm, YOLOv8-Pose, is utilized to estimate the position and orientation of surgical instruments in each individual camera view. These multi-view estimates enable the calculation of 3D poses of surgical tools, facilitating the rendering of a 3D surface model of the instruments, overlaid on the background scene, for real-time visualization. This study presents methods for adapting YOLOv8-Pose to the EasyVis2 system, including the development of a tailored training dataset. Experimental results demonstrate that, with an identical number of cameras, the new system improves 3D reconstruction accuracy and reduces computation time. Additionally, the adapted YOLOv8-Pose system shows high accuracy in 2D pose estimation.

Problem

• Real-time 3D visualization for laparoscopic surgery training
• Multi-view 3D pose estimation of surgical instruments
• Enhanced accuracy and speed in 3D reconstruction

Innovation

• Multi-camera trocar for 3D laparoscopic visualization
• YOLOv8-Pose deep network for instrument pose estimation
• Real-time 3D tool rendering with improved accuracy