LVLMs as inspectors: an agentic framework for category-level structural defect annotation

2025-10-01

AI Summary

To address the high cost, low efficiency, and labor-intensive nature of manual annotation for structural defects, this paper proposes ADPT—the first unsupervised intelligent agent framework for category-level structural defect annotation. ADPT integrates large vision-language models (VLMs), semantic pattern matching, recursive self-questioning-and-answering validation, and domain-specific prompt optimization. Leveraging semantic-driven automatic reasoning and iterative self-validation, it achieves end-to-end conversion from raw images to high-quality, category-level defect annotations. Evaluated on both balanced and imbalanced datasets, ADPT achieves annotation accuracies of 85%–98% and 80%–92% across four defect categories, respectively. It significantly improves annotation efficiency and cross-dataset generalization, establishing a scalable, zero-human-intervention paradigm for infrastructure safety inspection data construction.

Technology Category

• Computer Vision: Adversarial Attacks & Robustness
• Natural Language Processing: Safety and Robustness
• Machine Learning: Structured Learning

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Automated structural defect annotation is essential for ensuring infrastructure safety while minimizing the high costs and inefficiencies of manual labeling. A novel agentic annotation framework, Agent-based Defect Pattern Tagger (ADPT), is introduced that integrates Large Vision-Language Models (LVLMs) with a semantic pattern matching module and an iterative self-questioning refinement mechanism. By leveraging optimized domain-specific prompting and a recursive verification process, ADPT transforms raw visual data into high-quality, semantically labeled defect datasets without any manual supervision. Experimental results demonstrate that ADPT achieves up to 98% accuracy in distinguishing defective from non-defective images, and 85%-98% annotation accuracy across four defect categories under class-balanced settings, with 80%-92% accuracy on class-imbalanced datasets. The framework offers a scalable and cost-effective solution for high-fidelity dataset construction, providing strong support for downstream tasks such as transfer learning and domain adaptation in structural damage assessment.

Problem

• Automating structural defect annotation to reduce manual costs
• Converting raw visual data into labeled datasets without supervision
• Achieving high accuracy across balanced and imbalanced defect categories

Innovation

• LVLMs integrated with semantic pattern matching
• Iterative self-questioning refinement mechanism
• Domain-specific prompting with recursive verification