Multi-class segmentation of structural damage and pathological manifestations using YOLOv8 and Segment Anything Model

Abstract

Advances in computer vision have significantly improved bridge inspection by enabling precise damage detection and failure prediction. However, these techniques require costly datasets and specialized expertise. To overcome this, an approach combining YOLO object detection and SAM segmentation effectively identifies cracks, scaling, rust stains, exposed reinforcement, and efflorescence. Six models were fine-tuned, including the YOLOv8 architecture, three variations with modified detection layers for small, medium, and large damage, an optimized TensorRT version, and the new Yolov9-GELAN architecture. The YOLOv8l model achieved precision, recall, $mA{P}_{50}$, and $mA{P}_{50-95}$ of 0.946, 0.916, 0.951, and 0.892, respectively. The model’s outputs enhanced SAM-based instance segmentation, reducing uncertainties. A publicly available COCO-format dataset with 41,132 annotated images supports further research. This paper advances bridge inspection and construction by providing a robust model for multi-class object detection and instance segmentation of structural damages, with architectures tailored to detect small, medium, and large damages for more precise inspections.