Cross Branch Co-Attention Network multimodal models based on Raman and FTIR spectroscopy for diagnosis of multiple selected cancers

Abstract

The application of artificial intelligence (AI) in the medical field has brought unprecedented opportunities and challenges for early diagnosis and precision treatment of cancer. As complex multi-omics data in the medical field tends to be multimodal, a single type of data cannot provide enough information to support accurate diagnosis. Vibrational spectroscopy consists of Raman spectroscopy and FTIR spectroscopy, both of which can reflect the structural information of molecules and are used to detect the vibration and rotational energy levels of material molecules. However, the application of multimodal tasks in fusing vibrational spectroscopy is not comprehensive. In response to the above problems, this paper focuses on interactive multimodal fusion strategies to process and mine vibrational spectral information. A Cross Branch Co-Attention Network (CBCAN) is proposed to solve the problem of insufficient spectral fusion, and a spectral branch network and a collaborative attention network are constructed for collaborative information fusion. Finally, feature-level fusion is combined to achieve better sequential decision-making effects. Extensive experiments were conducted on cancer datasets and thyroid dysfunction binary classification datasets, with the corresponding sample numbers of 192 and 379, respectively. The research results show that compared with traditional deep learning algorithms and the latest related multimodal medical fusion methods, the proposed CBCAN classification model achieved 96.88 % accuracy, 93.61 % precision, 91.52 % sensitivity, 98.03 % specificity, 91.73 % F1 score and 99.75 % AUC value, respectively, with the best classification effect, providing a new method for rapid and non-invasive identification of multiple selected cancers, which has important reference value for the early diagnosis of cancer patients and helps to assist clinical diagnosis.