Are Synthetic Corruptions A Reliable Proxy For Real-World Corruptions?

2025-05-07

AI Summary

Real-world distribution shifts—such as weather and illumination variations—severely degrade the robustness of deep learning models. However, collecting diverse, real-world degraded data is prohibitively expensive, prompting widespread reliance on synthetic degradation; yet its fidelity in reflecting real-world degradation effects remains unclear. Method: We construct the largest cross-domain (real vs. synthetic) semantic segmentation corruption benchmark to date, built upon Cityscapes and other datasets using the CorruptIO toolkit. We systematically evaluate 12 corruption types across multiple models and metrics (mIoU, RankCorr). Contribution/Results: We discover, for the first time, a strong correlation (ρ = 0.89) between model performance under real and synthetic corruptions. We further propose a corruption-type-level correlation analysis framework to characterize the applicability boundaries of synthetic degradation. All evaluation code, protocols, and benchmarks are publicly released to advance standardized robustness assessment.

Technology Category

• Computer Vision: Adversarial Attacks & Robustness
• Machine Learning: Adversarial Learning & Robustness
• Natural Language Processing: Safety and Robustness

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

Deep learning (DL) models are widely used in real-world applications but remain vulnerable to distribution shifts, especially due to weather and lighting changes. Collecting diverse real-world data for testing the robustness of DL models is resource-intensive, making synthetic corruptions an attractive alternative for robustness testing. However, are synthetic corruptions a reliable proxy for real-world corruptions? To answer this, we conduct the largest benchmarking study on semantic segmentation models, comparing performance on real-world corruptions and synthetic corruptions datasets. Our results reveal a strong correlation in mean performance, supporting the use of synthetic corruptions for robustness evaluation. We further analyze corruption-specific correlations, providing key insights to understand when synthetic corruptions succeed in representing real-world corruptions. Open-source Code: https://github.com/shashankskagnihotri/benchmarking_robustness/tree/segmentation_david/semantic_segmentation

Problem

• Assessing reliability of synthetic corruptions for robustness testing
• Comparing DL model performance on real vs synthetic corruptions
• Identifying when synthetic corruptions mimic real-world effects

Innovation

• Largest benchmarking study on semantic segmentation models
• Compares real-world and synthetic corruption datasets
• Analyzes corruption-specific correlations for reliability insights