Cracks in the Foundation: A Civil Infrastructure Dataset to Challenge Vision Foundation Models

Unknown Date

AI Summary

This study addresses the challenges in structural health monitoring posed by data scarcity and model bias in pixel-level segmentation of civil infrastructure defects, such as cracks. To this end, the authors introduce CiF, the largest and most meticulously annotated instance segmentation dataset to date, comprising approximately 150,000 high-resolution images with five years of expert annotations. The work presents the first systematic evaluation of promptable foundation models, vision-language models, and specialized segmentation approaches in dense scene understanding within real-world built environments, revealing fundamental limitations in crack detection. Experiments demonstrate that current zero-shot foundation models exhibit substantially inadequate performance, while supervised task-specific models achieve a mean average precision (mAP) of only around 25%, establishing crack detection as a new benchmark challenge for visual understanding systems.

Abstract

Automated structural health monitoring is essential to prevent catastrophic infrastructure failures. Precise, pixel-level defect segmentation is needed to accurately assess structural integrity, but progress in defect segmentation for civil infrastructures has been held back by an extreme scarcity of data, which requires costly expert annotation. The need for data is accentuated by algorithmic hurdles intrinsic to the problem, including center-bias and the need to rely more on shape when inspecting nearly textureless building materials. To remove the bottleneck, we introduce Cracks in the Foundation (CiF), the largest and most detailed civil infrastructure (instance) segmentation dataset to date, comprising $\approx$150,000 high-resolution images meticulously curated over five years in collaboration with civil engineering experts. With the help of this unprecedented data source, we expose a blind spot of current visual AI: despite the advent of promptable Foundation Models (FMs) and Vision Language Models (VLMs), and despite the impressive abilities of today's specialised segmentation models, it turns out that dense image understanding in the built environment is nowhere near solved. Our evaluations indicate that even the most recent zero-shot FMs face significant challenges when deployed on real-world infrastructure and even the performance of specialised models with domain-specific supervision plateaus at $\approx$25% mAP. CiF establishes inspection of civil infrastructure, an elementary and seemingly easy perceptual task, as an open challenge that reveals fundamental weaknesses of present-day models trained predominantly on internet images, literally and figuratively highlighting cracks in the current foundation model paradigm.

Problem

• structural health monitoring
• defect segmentation
• civil infrastructure
• foundation models
• pixel-level understanding

Innovation

• foundation models
• infrastructure defect segmentation
• dataset scarcity
• zero-shot evaluation
• visual AI robustness