HET-RL: Multiple pulmonary disease diagnosis via hybrid efficient transformers based representation learning model using multi-modality data

Pulmonary diseases, encompassing conditions such as chronic bronchitis, emphysema, asthma, and pulmonary fibrosis, involve intricate pathophysiological mechanisms affecting the respiratory system, necessitating precise diagnosis and tailored therapeutic approaches. Timely and accurate diagnosis of pulmonary diseases is crucial as it enables early intervention, optimal management, and prevention of complications, thereby improving patient outcomes and quality of life. The scarcity of multi-modality datasets and challenges in accurate diagnosis underscore the complexities faced by deep learning models in achieving comprehensive pulmonary diagnoses, emphasizing the need for enhanced data diversity and algorithmic robustness in addressing diagnostic issues. To overcome these challenges, we have proposed a hybrid efficient transformer based on representation learning named as HET-RL model for accurate various pulmonary disease using multi-modality. Primarily in our work, we utilized multiple data including radiograph, chief complaints and clinical parameters for enhancing the efficiency of model performance. We have performed dual-level pre-processing such as denoising and normalization for amplifying data quality using Steered Filter (SF) and Min-Max normalization, respectively. Then, we proposed HET-RL model which encompasses of Inter-Attention Transformer (IAT) and Text Analyzer Transformer (TAT) for data analyzing. In which, appropriate features are analyzed and extracted from radiograph (CT scan) by IAT and TAT with representation learning (RL) encode the text and extract significant information from both chief complaint clinical parameters. Finally, the extracted features from hybrid transformers are fused by adapting Fusion Network and then pulmonary disease are classified into multiple classes. Incorporating diverse data sources, including results of laboratory test and patient demographic characteristics, our model demonstrated superior performance compared to non-unified multimodal and an image-only model diagnosis model. It exhibited a 12% and 9% improvement in identifying pulmonary disease. Multimodal hybrid transformer-based models hold promise for streamlining patient triaging and enhancing the clinical decision-making process.