MFCSA-CAT: a multimodal fusion method for cancer survival analysis based on cross-attention transformer

Abstract

Cancer diagnosis, prognosis, and therapeutic response predictions are based on data from various modalities, such as histology slides and molecular profiles from genomic data. In cancer clinical treatment, the technology of intelligent diagnosis for cancer patients has become an essential research domain with the rapid growth of various pathological data. In this work, we propose a multimodal fusion method for cancer survival analysis based on Cross-Attention Transformer. Compared to similar bimodal work, our work greatly reduces the number of parameters in the feature fusion model (our fusion model has 7625 parameters), and achieves the State-of-the-Art effect (81.85%) in bimodal cancer survival analysis task with histology images and genomic features data of Glioma cancer from TCGA database. （Previous bimodal Sota work in this task is Kronecker Product which achieves 81.40% with 170130 parameters）In addition, our experiments show that Cross-Attention can not only increase the correlation between the two modalities but also offer a better bimodal feature representation for the final fusion.