DIP-GS: Deep Image Prior For Gaussian Splatting Sparse View Recovery

2025-08-10

AI Summary

To address the poor reconstruction quality of 3D Gaussian Splatting (3DGS) under sparse-view conditions—characterized by few, incompletely distributed, and low-overlap input views—this paper proposes DIP-GS. It is the first work to integrate the pretraining-free Deep Image Prior (DIP) into the 3DGS optimization framework: a lightweight encoder-decoder network autonomously extracts internal structural priors directly from the input images, guiding coarse-to-fine joint optimization of Gaussian parameters. Crucially, DIP-GS entirely eliminates reliance on external models or prior datasets, leveraging only the intrinsic statistical regularities of the input images for unsupervised, robust 3D reconstruction. Experiments demonstrate that DIP-GS achieves state-of-the-art performance across diverse sparse-view configurations, notably improving geometric completeness and detail fidelity in extremely sparse settings (e.g., only 2–4 views) and low-overlap scenarios.

Technology Category

Computer Vision: 3D Computer VisionMachine Learning: Deep Generative Models & AutoencodersSearch and Optimization: Learning to Search

Application Category

Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphsUser Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendationWeb Mining and Content Analysis: Large pretrained models with web data

Abstract

3D Gaussian Splatting (3DGS) is a leading 3D scene reconstruction method, obtaining high-quality reconstruction with real-time rendering runtime performance. The main idea behind 3DGS is to represent the scene as a collection of 3D gaussians, while learning their parameters to fit the given views of the scene. While achieving superior performance in the presence of many views, 3DGS struggles with sparse view reconstruction, where the input views are sparse and do not fully cover the scene and have low overlaps. In this paper, we propose DIP-GS, a Deep Image Prior (DIP) 3DGS representation. By using the DIP prior, which utilizes internal structure and patterns, with coarse-to-fine manner, DIP-based 3DGS can operate in scenarios where vanilla 3DGS fails, such as sparse view recovery. Note that our approach does not use any pre-trained models such as generative models and depth estimation, but rather relies only on the input frames. Among such methods, DIP-GS obtains state-of-the-art (SOTA) competitive results on various sparse-view reconstruction tasks, demonstrating its capabilities.

Problem

• Improving sparse view reconstruction in 3D Gaussian Splatting
• Enhancing 3DGS performance without pre-trained models
• Achieving SOTA results with Deep Image Prior

Innovation

• Deep Image Prior enhances 3D Gaussian Splatting
• Coarse-to-fine DIP improves sparse view recovery
• No pre-trained models, only input frames used