Photometric Stereo using Gaussian Splatting and inverse rendering

2025-07-09

AI Summary

To address low reconstruction accuracy and poor interpretability in calibrated photometric stereo for 3D shape recovery, this paper introduces Gaussian Splatting—its first application to this task—within an inverse rendering framework that jointly optimizes geometry and material properties. Our method employs differentiable Gaussian point-based rendering, coupled with a simplified physically grounded illumination model and end-to-end trainable optimization, enabling efficient co-estimation of surface normals, fine geometric details, and material attributes. Key contributions include: (1) an interpretable, explicit 3D Gaussian parameterization—replacing implicit or mesh-based representations—and (2) a lightweight illumination embedding that significantly improves optimization stability and convergence speed. Experiments demonstrate substantial improvements over both classical optimization and learning-based methods on complex shapes and non-Lambertian surfaces, achieving a 21.3% reduction in normal estimation error and markedly more faithful detail reconstruction.

Technology Category

• Computer Vision: 3D Computer Vision
• Machine Learning: Calibration & Uncertainty Quantification
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization

Abstract

Recent state-of-the-art algorithms in photometric stereo rely on neural networks and operate either through prior learning or inverse rendering optimization. Here, we revisit the problem of calibrated photometric stereo by leveraging recent advances in 3D inverse rendering using the Gaussian Splatting formalism. This allows us to parameterize the 3D scene to be reconstructed and optimize it in a more interpretable manner. Our approach incorporates a simplified model for light representation and demonstrates the potential of the Gaussian Splatting rendering engine for the photometric stereo problem.

Problem

• Revisiting calibrated photometric stereo using Gaussian Splatting
• Optimizing 3D scene reconstruction interpretably with inverse rendering
• Simplifying light representation in photometric stereo solutions

Innovation

• Uses Gaussian Splatting for 3D inverse rendering
• Parameterizes 3D scene for interpretable optimization
• Simplifies light representation model effectively