Technology Category
Computer Vision: Motion & TrackingSearch and Optimization: Sampling/Simulation-based SearchPlanning, Routing, and Scheduling: Replanning and Plan Repair
Leveraging 2D Priors and SDF Guidance for Dynamic Urban Scene Rendering
🤖 AI Summary
Dynamic urban scene reconstruction typically relies heavily on LiDAR data, 3D instance segmentation, and ground-truth motion annotations—limiting scalability and practicality. To address this, we propose a weakly supervised framework that integrates 2D object-agnostic priors—monocular depth estimation and point trajectory tracking—with signed distance function (SDF) representation, embedded within the differentiable optimization pipeline of 3D Gaussian Splatting (3DGS). This yields a compact, geometrically accurate, and fully differentiable representation of dynamic objects without requiring explicit motion labels or 3D instance masks. Our method achieves high-fidelity novel-view synthesis directly from multi-view RGB video sequences. Incorporating sparse LiDAR further enhances geometric fidelity. Experiments demonstrate state-of-the-art rendering performance even in LiDAR-free settings, while enabling fine-grained editing capabilities such as scene decomposition and object replacement.
Technology Category
Application Category
📝 Abstract
Dynamic scene rendering and reconstruction play a crucial role in computer vision and augmented reality. Recent methods based on 3D Gaussian Splatting (3DGS), have enabled accurate modeling of dynamic urban scenes, but for urban scenes they require both camera and LiDAR data, ground-truth 3D segmentations and motion data in the form of tracklets or pre-defined object templates such as SMPL. In this work, we explore whether a combination of 2D object agnostic priors in the form of depth and point tracking coupled with a signed distance function (SDF) representation for dynamic objects can be used to relax some of these requirements. We present a novel approach that integrates Signed Distance Functions (SDFs) with 3D Gaussian Splatting (3DGS) to create a more robust object representation by harnessing the strengths of both methods. Our unified optimization framework enhances the geometric accuracy of 3D Gaussian splatting and improves deformation modeling within the SDF, resulting in a more adaptable and precise representation. We demonstrate that our method achieves state-of-the-art performance in rendering metrics even without LiDAR data on urban scenes. When incorporating LiDAR, our approach improved further in reconstructing and generating novel views across diverse object categories, without ground-truth 3D motion annotation. Additionally, our method enables various scene editing tasks, including scene decomposition, and scene composition.
Problem
Research questions and friction points this paper is trying to address.
Relaxing LiDAR and 3D annotation requirements for dynamic urban scene rendering
Integrating SDF with 3DGS to enhance geometric accuracy and deformation modeling
Achieving state-of-the-art rendering without ground-truth motion annotations
Innovation
Methods, ideas, or system contributions that make the work stand out.
Integrates SDF with 3D Gaussian Splatting
Uses 2D depth and point tracking priors
Enables editing without LiDAR or motion annotation
🔎 Similar Papers
No similar papers found.
