Free-DyGS: Camera-Pose-Free Scene Reconstruction for Dynamic Surgical Videos with Gaussian Splatting

2024-09-02

AI Summary

Addressing the challenge of 3D reconstruction in highly dynamic surgical scenes—where mobile endoscopes and deformable soft tissues coexist—this paper proposes the first free-pose dynamic surgical scene reconstruction method based on 3D Gaussian rasterization. Our approach tackles three core challenges: (1) robust pose-agnostic initialization via a sparse Gaussian regressor, eliminating prior pose assumptions; (2) joint optimization of 6-DoF camera trajectories and non-rigid deformation fields; and (3) a scene expansion mechanism coupled with retrospective deformation replay (RDR) to enhance temporal consistency and geometric fidelity. Unlike existing methods, ours requires no pre-calibration or fixed-camera setup. Evaluated on StereoMIS and Hamlyn datasets, it achieves state-of-the-art performance—improving PSNR by 2.1 dB while maintaining real-time inference (>25 FPS). This work establishes a deployable paradigm for dynamic 3D reconstruction in surgical navigation and medical education.

Technology Category

• Computer Vision: 3D Computer Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Planning, Routing, and Scheduling: Replanning and Plan Repair

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines

Abstract

High-fidelity reconstruction of surgical scene is a fundamentally crucial task to support many applications, such as intra-operative navigation and surgical education. However, most existing methods assume the ideal surgical scenarios - either focus on dynamic reconstruction with deforming tissue yet assuming a given fixed camera pose, or allow endoscope movement yet reconstructing the static scenes. In this paper, we target at a more realistic yet challenging setup - free-pose reconstruction with a moving camera for highly dynamic surgical scenes. Meanwhile, we take the first step to introduce Gaussian Splitting (GS) technique to tackle this challenging setting and propose a novel GS-based framework for fast reconstruction, termed extit{Free-DyGS}. Concretely, our model embraces a novel scene initialization in which a pre-trained Sparse Gaussian Regressor (SGR) can efficiently parameterize the initial attributes. For each subsequent frame, we propose to jointly optimize the deformation model and 6D camera poses in a frame-by-frame manner, easing training given the limited deformation differences between consecutive frames. A Scene Expansion scheme is followed to expand the GS model for the unseen regions introduced by the moving camera. Moreover, the framework is equipped with a novel Retrospective Deformation Recapitulation (RDR) strategy to preserve the entire-clip deformations throughout the frame-by-frame training scheme. The efficacy of the proposed Free-DyGS is substantiated through extensive experiments on two datasets: StereoMIS and Hamlyn datasets. The experimental outcomes underscore that Free-DyGS surpasses other advanced methods in both rendering accuracy and efficiency. Code will be available.

Problem

• Reconstructing dynamic surgical scenes with moving camera
• Integrating Gaussian Splatting for fast scene reconstruction
• Handling unseen regions and preserving deformation history

Innovation

• Uses Gaussian Splatting for dynamic scene reconstruction
• Jointly optimizes deformation model and camera poses
• Employs Retrospective Deformation Recapitulation strategy