Soft Shadow Diffusion (SSD): Physics-Inspired Learning for 3D Computational Periscopy

2026-01-18

European Conference on Computer Vision

AI Summary

This work overcomes the line-of-sight requirement in non-line-of-sight (NLOS) imaging by enabling high-fidelity 3D reconstruction of hidden scenes from a single ordinary photograph capturing subtle shadows on a wall. The authors propose a separable formulation of the light transport model that decomposes it into occluded and unoccluded components, leading to a separable nonlinear least squares (SNLLS) inverse problem. They solve this problem through a hybrid approach combining gradient-based optimization with a physically inspired Soft Shadow Diffusion (SSD) neural network. This method achieves, for the first time, single-image NLOS 3D reconstruction and demonstrates strong robustness to noise, ambient illumination, and unseen object categories in real-world scenarios, exhibiting excellent generalization capabilities.

Technology Category

Computer Vision: 3D Computer VisionSearch and Optimization: Learning to SearchIntelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for rankingGraph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphsEconomics, Online Markets and Human Computation: Architectures and workflows that use LLMs for crowd work

Abstract

Conventional imaging requires a line of sight to create accurate visual representations of a scene. In certain circumstances, however, obtaining a suitable line of sight may be impractical, dangerous, or even impossible. Non-line-of-sight (NLOS) imaging addresses this challenge by reconstructing the scene from indirect measurements. Recently, passive NLOS methods that use an ordinary photograph of the subtle shadow cast onto a visible wall by the hidden scene have gained interest. These methods are currently limited to 1D or low-resolution 2D color imaging or to localizing a hidden object whose shape is approximately known. Here, we generalize this class of methods and demonstrate a 3D reconstruction of a hidden scene from an ordinary NLOS photograph. To achieve this, we propose a novel reformulation of the light transport model that conveniently decomposes the hidden scene into \textit{light-occluding} and \textit{non-light-occluding} components to yield a separable non-linear least squares (SNLLS) inverse problem. We develop two solutions: A gradient-based optimization method and a physics-inspired neural network approach, which we call Soft Shadow diffusion (SSD). Despite the challenging ill-conditioned inverse problem encountered here, our approaches are effective on numerous 3D scenes in real experimental scenarios. Moreover, SSD is trained in simulation but generalizes well to unseen classes in simulation and real-world NLOS scenes. SSD also shows surprising robustness to noise and ambient illumination.

Problem

• Non-line-of-sight imaging
• 3D reconstruction
• Passive NLOS
• Hidden scene
• Soft Shadow Diffusion

Innovation

• Soft Shadow Diffusion
• non-line-of-sight imaging
• 3D reconstruction
• physics-inspired neural network
• separable non-linear least squares