NFL-BA: Improving Endoscopic SLAM with Near-Field Light Bundle Adjustment

2024-12-17

arXiv.org

AI Summary

Endoscopic SLAM faces challenges from near-field dynamic illumination and low-texture surfaces, which cause photometric bundle adjustment (BA) to fail. To address this, we propose a near-field illumination-aware BA loss function—the first to embed a physics-driven near-field lighting model (incorporating geometric relationships among light source, camera, and surface) into a differentiable SLAM optimization objective. Our method unifies neural implicit rendering with 3D Gaussian splatting (3DGS) to construct an end-to-end differentiable framework that explicitly models illumination variations while preserving geometric consistency. Evaluated on the C3VD colonoscopy dataset and real clinical videos, our approach significantly improves pose estimation accuracy and dense reconstruction robustness, achieving state-of-the-art performance. Moreover, it demonstrates strong generalizability to neural-rendering-based SLAM systems.

Technology Category

• Computer Vision: 3D Computer Vision
• Intelligent Robots: State Estimation
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

Simultaneous Localization And Mapping (SLAM) from endoscopy videos can enable autonomous navigation, guidance to unsurveyed regions, blindspot detections, and 3D visualizations, which can significantly improve patient outcomes and endoscopy experience for both physicians and patients. Existing dense SLAM algorithms often assume distant and static lighting and optimize scene geometry and camera parameters by minimizing a photometric rendering loss, often called Photometric Bundle Adjustment. However, endoscopy videos exhibit dynamic near-field lighting due to the co-located light and camera moving extremely close to the surface. In addition, low texture surfaces in endoscopy videos cause photometric bundle adjustment of the existing SLAM frameworks to perform poorly compared to indoor/outdoor scenes. To mitigate this problem, we introduce Near-Field Lighting Bundle Adjustment Loss (NFL-BA) which explicitly models near-field lighting as a part of Bundle Adjustment loss and enables better performance for low texture surfaces. Our proposed NFL-BA can be applied to any neural-rendering based SLAM framework. We show that by replacing traditional photometric bundle adjustment loss with our proposed NFL-BA results in improvement, using neural implicit SLAM and 3DGS SLAMs. In addition to producing state-of-the-art tracking and mapping results on colonoscopy C3VD dataset we also show improvement on real colonoscopy videos. See results at https://asdunnbe.github.io/NFL-BA/

Problem

• Improves SLAM in endoscopy with near-field lighting modeling
• Addresses poor performance on low texture surfaces in endoscopy
• Enhances tracking and mapping in real colonoscopy videos

Innovation

• Introduces Near-Field Lighting Bundle Adjustment Loss
• Improves SLAM for low texture endoscopic surfaces
• Applicable to neural-rendering based SLAM frameworks