Sparse-DeRF: Deblurred Neural Radiance Fields from Sparse View

2024-07-09

arXiv.org

AI Summary

This work addresses the ill-posed problem of joint optimization of spatially varying blur kernels and neural radiance fields (NeRFs) for 3D deblurring reconstruction from extremely sparse (2–6 views), blurry input images. To mitigate this ill-posedness, we propose DeRF—a novel framework incorporating three complementary regularizations: (i) an implicit-surface-based geometric prior to enforce structural consistency; (ii) modulated gradient scaling to adaptively enhance scene structure awareness; and (iii) VGG-based perceptual distillation to recover fine details lost in blur. DeRF integrates differentiable NeRF rendering, learnable spatially varying blur kernel modeling, and statistics-driven implicit ray sampling. Experiments demonstrate that DeRF effectively suppresses overfitting under extreme view sparsity, achieving state-of-the-art performance across PSNR, SSIM, and LPIPS metrics, and enabling high-fidelity, robust 3D deblurring reconstruction.

Technology Category

• Computer Vision: 3D Computer Vision
• Machine Learning: Kernel Methods
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Recent studies construct deblurred neural radiance fields~(DeRF) using dozens of blurry images, which are not practical scenarios if only a limited number of blurry images are available. This paper focuses on constructing DeRF from sparse-view for more pragmatic real-world scenarios. As observed in our experiments, establishing DeRF from sparse views proves to be a more challenging problem due to the inherent complexity arising from the simultaneous optimization of blur kernels and NeRF from sparse view. Sparse-DeRF successfully regularizes the complicated joint optimization, presenting alleviated overfitting artifacts and enhanced quality on radiance fields. The regularization consists of three key components: Surface smoothness, helps the model accurately predict the scene structure utilizing unseen and additional hidden rays derived from the blur kernel based on statistical tendencies of real-world; Modulated gradient scaling, helps the model adjust the amount of the backpropagated gradient according to the arrangements of scene objects; Perceptual distillation improves the perceptual quality by overcoming the ill-posed multi-view inconsistency of image deblurring and distilling the pre-deblurred information, compensating for the lack of clean information in blurry images. We demonstrate the effectiveness of the Sparse-DeRF with extensive quantitative and qualitative experimental results by training DeRF from 2-view, 4-view, and 6-view blurry images.

Problem

• Constructing DeRF from sparse-view blurry images
• Joint optimization of blur kernels and NeRF
• Regularizing overfitting and enhancing radiance field quality

Innovation

• Surface smoothness regularizes scene structure prediction
• Modulated gradient scaling adjusts backpropagated gradients
• Perceptual distillation enhances multi-view deblurring quality