Multimodal deep learning-based automatic generation of repair proposals for steel bridge shallow damage

Abstract

As bridges age, manual repair decision-making methods struggle to meet growing maintenance demands. This paper develops AI systems that can imitate experts' decision processes by mining implicit relationships between bridge damage images and corresponding repair proposals. A multimodal deep learning-based end-to-end decision-making method is proposed to extract and map features of bridge damage images and repair proposal texts, automating damage repair proposal generation. The model is trained and validated using a dataset from historical inspection reports. The model's image feature extraction is evaluated using Class Activation Mapping (CAM), while text generation achieved BLEU-1 to BLEU-4 scores of 0.76, 0.743, 0.712, and 0.705, respectively, with 82 % accuracy in human evaluation. The results indicate the model's effectiveness in handling complex image features and generating long text, addressing challenges in automated bridge repair decision-making.