Problem
Research questions and friction points this paper is trying to address.
3D cityscape animation
visual accuracy
lighting and shadow representation
UrbanIR: Large-Scale Urban Scene Inverse Rendering from a Single Video
🤖 AI Summary
Existing single-video inverse rendering methods for large-scale urban scenes suffer from roof geometry distortion, floating objects, and inconsistent shadows and illumination, hindering accurate reconstruction of building geometry, material properties, and visibility. This paper introduces the first neural inverse rendering framework tailored to wide-baseline vehicle-mounted video sequences. It jointly optimizes scene geometry, albedo, visibility, and directional sunlight/sky illumination, incorporating a novel inverse graphics loss and shadow consistency regularization. For the first time, our method achieves high-fidelity reconstruction of original scene shadow volumes, substantially suppressing floating artifacts and roof geometry errors. Evaluated on real-world urban videos, it enables free-viewpoint rendering, relighting, and nighttime scene synthesis. Quantitative and qualitative evaluations demonstrate superior geometric and material reconstruction accuracy, as well as enhanced visual quality, outperforming current state-of-the-art approaches.
📝 Abstract
We present UrbanIR (Urban Scene Inverse Rendering), a new inverse graphics model that enables realistic, free-viewpoint renderings of scenes under various lighting conditions with a single video. It accurately infers shape, albedo, visibility, and sun and sky illumination from wide-baseline videos, such as those from car-mounted cameras, differing from NeRF's dense view settings. In this context, standard methods often yield subpar geometry and material estimates, such as inaccurate roof representations and numerous 'floaters'. UrbanIR addresses these issues with novel losses that reduce errors in inverse graphics inference and rendering artifacts. Its techniques allow for precise shadow volume estimation in the original scene. The model's outputs support controllable editing, enabling photorealistic free-viewpoint renderings of night simulations, relit scenes, and inserted objects, marking a significant improvement over existing state-of-the-art methods.
Innovation
Methods, ideas, or system contributions that make the work stand out.
UrbanIR
Single-video Urban Scene Animation
NeRF Advancement
🔎 Similar Papers
No similar papers found.
