Question-answering framework for building codes using fine-tuned and distilled pre-trained transformer models

Abstract

Building code compliance checking is considered a bottleneck in construction projects, which calls for a novel approach to building code query and information retrieval. To address this research gap, the paper presents a question and answering framework comprising: (1) a ‘retriever’ for efficient context retrieval from building codes in response to an inquiry, and (2) a ‘reader’ for precise context interpretation and answer generation. The ‘retriever’, based on the BM25 algorithm, achieved a top-1 precision, recall, and F1-score of 0.95, 0.95, and 0.95, and a top-5 precision, recall, and F1-score of 0.97, 1.00, and 0.99, respectively. The ‘reader’, utilizing the transformer-based “xlm-roberta-base-squad2-distilled” model, achieved a top-4 accuracy of 0.95 and a top-1 F1-score of 0.84. A fine-tuning and model distillation process was used and shown to provide high performance on limited amount of training data, overcoming a common barrier in the development of domain-specific (e.g., construction) deep learning models.