Automating construction safety inspections using a multi-modal vision-language RAG framework

2025-10-05

AI Summary

Traditional construction safety inspections rely heavily on manual labor, resulting in low efficiency; meanwhile, existing large vision-language models (LVLMs) suffer from vague responses, limited multimodal input support, and hallucination, while pure large language model (LLM) approaches are hindered by scarce domain-specific training data and poor real-time adaptability. To address these challenges, this paper proposes SiteShield—the first multimodal retrieval-augmented generation (RAG) framework specifically designed for construction safety. SiteShield jointly processes image, video, and audio inputs, synergizing vision-language understanding with dynamic external knowledge retrieval to substantially mitigate hallucination and enhance both reasoning accuracy and temporal responsiveness. Evaluated on a real-world construction site dataset, SiteShield achieves an F1-score of 0.82, precision of 76%, recall of 96%, and a Hamming loss of only 0.04—outperforming all unimodal baselines across all metrics.

Technology Category

• Computer Vision: Multi-modal Vision
• Natural Language Processing: Safety and Robustness
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Security and Privacy: Large-scale security measurements
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Conventional construction safety inspection methods are often inefficient as they require navigating through large volume of information. Recent advances in large vision-language models (LVLMs) provide opportunities to automate safety inspections through enhanced visual and linguistic understanding. However, existing applications face limitations including irrelevant or unspecific responses, restricted modal inputs and hallucinations. Utilisation of Large Language Models (LLMs) for this purpose is constrained by availability of training data and frequently lack real-time adaptability. This study introduces SiteShield, a multi-modal LVLM-based Retrieval-Augmented Generation (RAG) framework for automating construction safety inspection reports by integrating visual and audio inputs. Using real-world data, SiteShield outperformed unimodal LLMs without RAG with an F1 score of 0.82, hamming loss of 0.04, precision of 0.76, and recall of 0.96. The findings indicate that SiteShield offers a novel pathway to enhance information retrieval and efficiency in generating safety reports.

Problem

• Automating construction safety inspections using multi-modal vision-language framework
• Addressing limitations of irrelevant responses and restricted modal inputs
• Enhancing efficiency in generating safety inspection reports through RAG

Innovation

• Multi-modal vision-language RAG framework for safety inspections
• Integrates visual and audio inputs for automated reporting
• Outperforms unimodal LLMs with enhanced retrieval efficiency