IDDR-NGP:Incorporating Detectors for Distractors Removal with Instant Neural Radiance Field

2023-10-26

ACM Multimedia

AI Summary

This work proposes the first unified framework capable of removing diverse real-world and synthetic distractors—such as snowflakes, confetti, and fallen leaves—from 3D scenes, addressing the limitation of existing methods that handle only a single distractor type. The approach integrates a 2D distractor detector with an Instant-NGP-based implicit 3D representation and is trained end-to-end using a joint optimization strategy that combines LPIPS loss with a novel multi-view compensation loss (MVCL). Additionally, the authors introduce the first benchmark dataset specifically designed for distractor removal in implicit 3D representations. Experimental results demonstrate that the method exhibits strong robustness across multiple distractor types, achieves snow removal performance on par with current state-of-the-art methods, and effectively recovers high-quality 3D scenes.

Technology Category

• Computer Vision: 3D Computer Vision
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Web data integration and cleaning

Abstract

This paper presents the first unified distractor removal method, named IDDR-NGP, which directly operates on Instant-NPG. The method is able to remove a wide range of distractors in 3D scenes, such as snowflakes, confetti, defoliation and petals, whereas existing methods usually focus on a specific type of distractors. By incorporating implicit 3D representations with 2D detectors, we demonstrate that it is possible to efficiently restore 3D scenes from multiple corrupted images. We design the learned perceptual image patch similarity~( LPIPS) loss and the multi-view compensation loss (MVCL) to jointly optimize the rendering results of IDDR-NGP, which could aggregate information from multi-view corrupted images. All of them can be trained in an end-to-end manner to synthesize high-quality 3D scenes. To support the research on distractors removal in implicit 3D representations, we build a new benchmark dataset that consists of both synthetic and real-world distractors. To validate the effectiveness and robustness of IDDR-NGP, we provide a wide range of distractors with corresponding annotated labels added to both realistic and synthetic scenesExtensive experimental results demonstrate the effectiveness and robustness of IDDR-NGP in removing multiple types of distractors. In addition, our approach achieves results comparable with the existing SOTA desnow methods and is capable of accurately removing both realistic and synthetic distractors.

Problem

• distractor removal
• 3D scene reconstruction
• implicit 3D representations
• multi-view corrupted images
• Instant-NGP

Innovation

• distractor removal
• Instant-NGP
• implicit 3D representation
• multi-view compensation
• end-to-end training