An integrated approach for automatic safety inspection in construction: Domain knowledge with multimodal large language model

Abstract

This research addresses the challenge of dynamically integrating visual and textual data in construction safety inspections while enhancing adaptability to new safety hazards and ensuring faithful interpretation of safety rules. We propose a novel approach that seamlessly combines multi-modal techniques with domain knowledge, advancing beyond current methods that often struggle with multi-modal understanding and adaptation to new safety hazards. Our approach consists of three key components: (1) a fine-tuned multi-modal LLM for visual and textual processing, (2) a domain knowledge base for evolving safety standards adaptability and output faithfulness, and (3) a multi-step reasoning engine to tackle complex safety inspection tasks. We validate our approach using on-site data from Wuhan subway construction sites, demonstrating its capability to perform moderately accurate (0.57 hazard identification correctness), contextually relevant (0.96 on task relevancy), and faithful safety assessments (0.95 and 0.99 on reasoning faithfulness). The results suggest promising performance in construction scene perception, as well as textual analysis and reasoning. This approach represents an advancement in automatic construction safety inspection and contributes to the broader discourse on formalizing multi-modal processing of construction data, offering insights into creating more flexible and comprehensive safety management systems.