A collaborative multi-task model for immunohistochemical molecular sub-types of multi-modal breast cancer MRI images

Clinically, personalized treatment developed based on the immunohistochemical (IHC) molecular sub-types of breast cancer can enhance long-term survival rates. Nevertheless, IHC, as an invasive detection method, may pose some risks of tumor metastasis caused by puncture. This work propose a collaborative multi-task model based on multi-modal data. Firstly, a dual-stream learning network based on Swin Transformer is employed to extract features from both DCE and T1WI images. Specifically, an Shared Representation (SR) module extracts shared representations, while an Enhancement of Unique features (EU) module enhances specific features. Subsequently, a multi-path classification network is constructed, which comprehensively considers the MRI image features, lesion location, and morphological features. Comprehensive experiments using clinical MRI images show the proposed method outperforms state-of-the-art, with an accuracy of 85.1%, sensitivity of 84.0%, specificity of 95.1%, and an F1 score of 83.6%.