SILICA: Repurposing Diffusion Priors for Joint Glass Segmentation and Depth Estimation

2026-07-27

AI Summary

Transparent surfaces cause depth sensors to fail, leading to errors in 3D mapping and navigation; however, existing learning-based approaches struggle to generalize due to the scarcity of real-world glass depth annotations. This work proposes SILICA, a unified framework that, for the first time, leverages the prior knowledge of text-to-image diffusion models for transparent surface perception. SILICA jointly predicts glass segmentation and glass-aware depth, exploiting their mutual interaction to construct a spatial hierarchical representation—enabling zero-shot transfer without requiring paired real glass depth labels. By harnessing diffusion priors to correct sensor depth errors, the method significantly outperforms current state-of-the-art approaches by nearly 20% across diverse unseen environments and establishes a new benchmark on the newly introduced Mirage18k dataset for transparent surface perception.

Abstract

Standard depth sensors systematically fail on transparent surfaces, creating corrupted 3D maps and severe navigation hazards. While specialized hardware sensors can detect glass, they lack modularity and have extensive hardware dependencies. Consequently, learning-based monocular depth estimation has emerged as a compelling alternative. However, domain-specific glass-aware monocular depth estimators struggle with unfamiliar indoor layouts; restricted by the severe scarcity of real-world glass depth annotations, they fail to generalize zero-shot to new settings. This motivates us to explore whether the extensive priors of text-to-image diffusion models can enable generalizable perception of transparent surfaces. We introduce SILICA, a unified pipeline leveraging these priors to jointly predict glass segmentation and glass-aware depth. This mutual information exchange establishes a robust spatial hierarchy, entirely eliminating the need for paired real-world glass depth annotations. Subsequently, we use the predicted segmentation mask to explicitly filter incorrect glass depth points from standard sensors, recovering accurate metric glass depth for downstream 3D mapping and autonomous collision avoidance. Supported by our novel Mirage 18k dataset, extensive experiments demonstrate that SILICA achieves remarkable zero-shot transfer across diverse, unseen environments, outperforming state-of-the-art models by almost 20% and setting a new benchmark for transparent surface perception.

Problem

• transparent surface
• depth estimation
• glass segmentation
• zero-shot generalization
• monocular depth

Innovation

• diffusion priors
• glass segmentation
• monocular depth estimation
• zero-shot transfer
• transparent surface perception