Arbitrary Volumetric Refocusing of Dense and Sparse Light Fields

2025-02-26

AI Summary

Existing light field refocusing methods are constrained to single-depth-layer processing, limiting simultaneous sharpening and defocusing of multiple arbitrary planar or volumetric regions within the same depth range; moreover, they suffer from ghosting artifacts under sparse light field sampling. This paper proposes a post-processing refocusing framework compatible with both dense and sparse light fields. It introduces voxel-wise refocusing with pixel-level independent displacement control—the first such approach—designs a U-Net-based ghosting suppression module to mitigate inherent artifacts in sparse displacement-and-sum operations, and develops an improved shift-and-sum algorithm with pixel-dependent displacements integrated into an end-to-end jointly optimized reconstruction network. Experiments demonstrate that, at only 20% sampling rate, the method achieves SSIM > 0.91, significantly enhancing refocusing fidelity and flexibility for non-planar objects.

Technology Category

• Computer Vision: 3D Computer Vision
• Search and Optimization: Learning to Search
• Machine Learning: Multimodal Learning

Application Category

• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

A four-dimensional light field (LF) captures both textural and geometrical information of a scene in contrast to a two-dimensional image that captures only the textural information of a scene. Post-capture refocusing is an exciting application of LFs enabled by the geometric information captured. Previously proposed LF refocusing methods are mostly limited to the refocusing of single planar or volumetric region of a scene corresponding to a depth range and cannot simultaneously generate in-focus and out-of-focus regions having the same depth range. In this paper, we propose an end-to-end pipeline to simultaneously refocus multiple arbitrary planar or volumetric regions of a dense or a sparse LF. We employ pixel-dependent shifts with the typical shift-and-sum method to refocus an LF. The pixel-dependent shifts enables to refocus each pixel of an LF independently. For sparse LFs, the shift-and-sum method introduces ghosting artifacts due to the spatial undersampling. We employ a deep learning model based on U-Net architecture to almost completely eliminate the ghosting artifacts. The experimental results obtained with several LF datasets confirm the effectiveness of the proposed method. In particular, sparse LFs refocused with the proposed method archive structural similarity index higher than 0.9 despite having only 20% of data compared to dense LFs.

Problem

• Refocus multiple arbitrary planar or volumetric regions
• Eliminate ghosting artifacts in sparse light fields
• Achieve high structural similarity with sparse data

Innovation

• End-to-end pipeline for light field refocusing
• Pixel-dependent shifts for independent refocusing
• U-Net architecture to eliminate ghosting artifacts