DOF-GS: Adjustable Depth-of-Field 3D Gaussian Splatting for Post-Capture Refocusing, Defocus Rendering and Blur Removal

2024-05-27

AI Summary

Existing 3D Gaussian Splatting (3DGS) methods rely on the pinhole camera model, which cannot represent defocus blur—limiting applications such as post-capture refocusing, all-in-focus rendering, and deblurring. This work proposes the first differentiable depth-of-field rendering framework for finite-aperture cameras, enabling explicit modeling of adjustable aperture and focus distance within 3DGS for the first time. It supports 3D reconstruction and post-capture refocusing from multi-view blurry images. A novel learnable Circle-of-Confusion (CoC)-aware optimization mechanism is introduced to identify in-focus regions and jointly optimize geometric and blur cues. The method requires no camera calibration. Experiments demonstrate significant improvements in refocusing accuracy, controllability of bokeh effects, and all-in-focus image synthesis quality, outperforming state-of-the-art 3DGS approaches across multiple tasks.

Technology Category

• Computer Vision: 3D Computer Vision
• Search and Optimization: Sampling/Simulation-based Search
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

3D Gaussian Splatting (3DGS) techniques have recently enabled high-quality 3D scene reconstruction and real-time novel view synthesis. These approaches, however, are limited by the pinhole camera model and lack effective modeling of defocus effects. Departing from this, we introduce DOF-GS--a new 3DGS-based framework with a finite-aperture camera model and explicit, differentiable defocus rendering, enabling it to function as a post-capture control tool. By training with multi-view images with moderate defocus blur, DOF-GS learns inherent camera characteristics and reconstructs sharp details of the underlying scene, particularly, enabling rendering of varying DOF effects through on-demand aperture and focal distance control, post-capture and optimization. Additionally, our framework extracts circle-of-confusion cues during optimization to identify in-focus regions in input views, enhancing the reconstructed 3D scene details. Experimental results demonstrate that DOF-GS supports post-capture refocusing, adjustable defocus and high-quality all-in-focus rendering, from multi-view images with uncalibrated defocus blur.

Problem

• Enables post-capture adjustable depth-of-field effects
• Reconstructs sharp scene details from defocused images
• Supports uncalibrated blur removal and refocusing

Innovation

• Finite-aperture camera model for defocus rendering
• Differentiable rendering for post-capture control
• Circle-of-confusion cues enhance 3D scene details