Technology Category
Computer Vision: Motion & TrackingMachine Learning: Unsupervised & Self-Supervised LearningKnowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
Object Learning and Robust 3D Reconstruction
🤖 AI Summary
This work addresses motion-induced artifacts in unsupervised 3D reconstruction of dynamic scenes caused by moving foreground objects. To tackle this, we propose a novel method that jointly models unsupervised 2D object decomposition and 3D geometric consistency. Our approach introduces FlowCapsules—a flow-guided network for unsupervised foreground segmentation—and a transient-object-mask-driven robust optimization kernel that detects and excludes dynamic objects under multi-view geometric consistency constraints. This kernel subsequently guides weighted bundle adjustment and NeRF training. Crucially, our method is the first to jointly learn object-centric representations and scene-level geometry without any annotations or controlled capture conditions. Experiments demonstrate significant improvements in both SfM and NeRF reconstruction accuracy on casually captured dynamic scenes, effectively eliminating motion artifacts. The framework advances explicit object-aware 3D understanding in open-world vision applications.
Technology Category
Application Category
📝 Abstract
In this thesis we discuss architectural designs and training methods for a neural network to have the ability of dissecting an image into objects of interest without supervision. The main challenge in 2D unsupervised object segmentation is distinguishing between foreground objects of interest and background. FlowCapsules uses motion as a cue for the objects of interest in 2D scenarios. The last part of this thesis focuses on 3D applications where the goal is detecting and removal of the object of interest from the input images. In these tasks, we leverage the geometric consistency of scenes in 3D to detect the inconsistent dynamic objects. Our transient object masks are then used for designing robust optimization kernels to improve 3D modelling in a casual capture setup. One of our goals in this thesis is to show the merits of unsupervised object based approaches in computer vision. Furthermore, we suggest possible directions for defining objects of interest or foreground objects without requiring supervision. Our hope is to motivate and excite the community into further exploring explicit object representations in image understanding tasks.
Problem
Research questions and friction points this paper is trying to address.
Unsupervised 2D object segmentation using motion cues
3D dynamic object detection via geometric consistency
Robust 3D modeling with transient object masks
Innovation
Methods, ideas, or system contributions that make the work stand out.
Unsupervised object segmentation using motion cues
3D geometric consistency for dynamic object detection
Robust optimization kernels for 3D modeling
🔎 Similar Papers
2024-02-12arXiv.orgCitations: 3
