Toward Reliable RGB-D Semantic Segmentation: Handling Missing Modalities via Condition Dropout

2026-07-22

AI Summary

This work addresses the performance degradation of RGB-D semantic segmentation models under real-world deployment scenarios where sensor failures or occlusions often lead to missing modalities. To enhance robustness against such modality loss, the authors propose Condition Dropout (ConD), a two-stage training strategy that fine-tunes a pre-trained model by randomly simulating inputs with complete data, missing RGB, or missing depth modalities. ConD employs a combination of freezing the original encoder, duplicating the encoder, and injecting zero-initialized features, thereby preserving performance on complete inputs while significantly improving segmentation accuracy under missing-modality conditions. Extensive experiments on NYU-Depth V2 and SUN RGB-D datasets demonstrate the effectiveness of the proposed approach, which also yields slight performance gains even when both modalities are fully available.

Abstract

RGB-D semantic segmentation has achieved remarkable progress, yet most models assume that RGB and depth are always available. In practice, failures or occlusions of surveillance sensors often remove one modality. Although RGB or depth alone can contain sufficient cues, models trained only on full-modality inputs fail to exploit the remaining modality once one is missing, causing severe degradation. We tackle this issue with a simple continued-training paradigm, \emph{Condition Dropout (ConD)}, which mitigates degradation while preserving full-modality accuracy. Starting from a pretrained RGB-D model, ConD adds a second stage that randomly simulates complete, RGB-missing, and depth-missing inputs, freezes the original encoders, and trains copied encoders with zero-initialized feature injection. Experiments on NYU-Depth V2 and SUN RGB-D show that ConD improves robustness under missing modalities and even yields slight gains when modalities are complete. Our code will be made publicly available upon acceptance.

Problem

• RGB-D semantic segmentation
• missing modality
• sensor failure
• occlusion
• model robustness

Innovation

• Condition Dropout
• RGB-D semantic segmentation
• modality missing
• robustness
• feature injection