Tracking-Aware Deformation Field Estimation for Non-rigid 3D Reconstruction in Robotic Surgeries

2025-03-04

AI Summary

Addressing the challenge of real-time 3D perception of non-rigid soft-tissue deformation in robotic laparoscopic surgery, this paper proposes the Tracking-Aware Deformation Field (TADF) framework. TADF is the first method to jointly optimize vision foundation model–driven keypoint tracking with neural implicit surface reconstruction (NeRF-like), thereby unifying 2D keypoint trajectory estimation and 3D deformation field modeling. This integration enables robust, 2D-guided 3D deformation estimation and synchronized dynamic mesh reconstruction. Evaluated on two public surgical datasets, TADF significantly outperforms existing neural 3D reconstruction approaches in deformation estimation accuracy while maintaining real-time inference capability and high geometric fidelity. The framework establishes a novel paradigm for minimally invasive surgical navigation—delivering high-precision, interpretable, and geometrically consistent soft-tissue deformation perception.

Technology Category

• Computer Vision: 3D Computer Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Natural Language Processing: Safety and Robustness

Application Category

• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Minimally invasive procedures have been advanced rapidly by the robotic laparoscopic surgery. The latter greatly assists surgeons in sophisticated and precise operations with reduced invasiveness. Nevertheless, it is still safety critical to be aware of even the least tissue deformation during instrument-tissue interactions, especially in 3D space. To address this, recent works rely on NeRF to render 2D videos from different perspectives and eliminate occlusions. However, most of the methods fail to predict the accurate 3D shapes and associated deformation estimates robustly. Differently, we propose Tracking-Aware Deformation Field (TADF), a novel framework which reconstructs the 3D mesh along with the 3D tissue deformation simultaneously. It first tracks the key points of soft tissue by a foundation vision model, providing an accurate 2D deformation field. Then, the 2D deformation field is smoothly incorporated with a neural implicit reconstruction network to obtain tissue deformation in the 3D space. Finally, we experimentally demonstrate that the proposed method provides more accurate deformation estimation compared with other 3D neural reconstruction methods in two public datasets.

Problem

• Estimating 3D tissue deformation in robotic surgeries.
• Reconstructing 3D mesh with accurate deformation fields.
• Improving 3D neural reconstruction accuracy in surgery.

Innovation

• TADF framework for 3D mesh and deformation reconstruction
• Key point tracking using foundation vision model
• Neural implicit network for 3D deformation estimation