GS-ID: Illumination Decomposition on Gaussian Splatting via Diffusion Prior and Parametric Light Source Optimization

2024-08-16

arXiv.org

AI Summary

The strong coupling among geometry, material, and illumination in Gaussian Splatting (GS) scenes leads to an ill-posed illumination decomposition problem. To address this, we propose the first illumination disentanglement framework that jointly optimizes parametric light sources while incorporating intrinsic image diffusion priors. Our method explicitly decouples ambient and direct illumination: ambient lighting is modeled via a learnable spherical environment map, while multiple directional lights are represented as Spherical Gaussians (SGs). Illumination-aware deferred rendering ensures physically consistent relighting. Evaluated on standard benchmarks, our approach achieves state-of-the-art performance in illumination decomposition accuracy, geometric reconstruction quality, and novel-view synthesis fidelity. It enables efficient, controllable, and photorealistic real-time relighting—significantly enhancing both editability and rendering authenticity of GS-based scenes.

Technology Category

Computer Vision: Diffusion Models for VisionMachine Learning: Learning with ManifoldsSearch and Optimization: Mixed Discrete/Continuous Search

Application Category

Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for rankingGraph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphsSystems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

We present GS-ID, a novel framework for illumination decomposition on Gaussian Splatting, achieving photorealistic novel view synthesis and intuitive light editing. Illumination decomposition is an ill-posed problem facing three main challenges: 1) priors for geometry and material are often lacking; 2) complex illumination conditions involve multiple unknown light sources; and 3) calculating surface shading with numerous light sources is computationally expensive. To address these challenges, we first introduce intrinsic diffusion priors to estimate the attributes for physically based rendering. Then we divide the illumination into environmental and direct components for joint optimization. Last, we employ deferred rendering to reduce the computational load. Our framework uses a learnable environment map and Spherical Gaussians (SGs) to represent light sources parametrically, therefore enabling controllable and photorealistic relighting on Gaussian Splatting. Extensive experiments and applications demonstrate that GS-ID produces state-of-the-art illumination decomposition results while achieving better geometry reconstruction and rendering performance.

Problem

• Disentangle geometry, material, lighting in Gaussian Splatting
• Handle non-Lambertian conditions with specularities and shadows
• Improve inverse rendering and relighting via illumination decomposition

Innovation

• Adaptive light aggregation for illumination decomposition
• Diffusion-based material priors for scene editing
• Anisotropic spherical Gaussian mixtures for local lighting