SteeredMarigold: Steering Diffusion Towards Depth Completion of Largely Incomplete Depth Maps

2024-09-16

arXiv.org

AI Summary

Conventional depth completion methods fail when RGB-D sensors produce severely sparse depth maps—i.e., large-scale depth missingness—due to their reliance on dense initial depth coverage. Method: This paper proposes a zero-shot, training-free diffusion model–guided approach that directly conditions a pre-trained diffusion model (Marigold) on sparse depth points. By jointly modeling sparse depth constraints and RGB priors, the method enables end-to-end generation of metrically accurate, dense depth maps. Contribution/Results: The approach eliminates dependence on initial depth coverage, substantially improving robustness under extreme sparsity. It achieves state-of-the-art performance on the NYUv2 severe missingness benchmark and releases open-source code.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Search and Optimization: Learning to Search

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Even if the depth maps captured by RGB-D sensors deployed in real environments are often characterized by large areas missing valid depth measurements, the vast majority of depth completion methods still assumes depth values covering all areas of the scene. To address this limitation, we introduce SteeredMarigold, a training-free, zero-shot depth completion method capable of producing metric dense depth, even for largely incomplete depth maps. SteeredMarigold achieves this by using the available sparse depth points as conditions to steer a denoising diffusion probabilistic model. Our method outperforms relevant top-performing methods on the NYUv2 dataset, in tests where no depth was provided for a large area, achieving state-of-art performance and exhibiting remarkable robustness against depth map incompleteness. Our source code is publicly available at https://steeredmarigold.github.io.

Problem

• Completing incomplete depth maps
• Using sparse depth points
• Achieving state-of-art performance

Innovation

• Training-free depth completion
• Steers diffusion with sparse depth
• Outperforms on NYUv2 dataset