Clinical knowledge integrated multi-task learning network for breast tumor segmentation and pathological complete response prediction

Abstract

The accurate segmentation of breast tumors helps determine the boundaries and size of the tumor, providing crucial information for subsequent treatment planning. It also enables a more precise characterization of the tumor, which can be used to predict the patient’s response to neoadjuvant chemotherapy. Existing methodologies predominantly rely on single-task learning, overlooking the potential inter-task correlations inherent in multi-task learning. Moreover, the available clinical knowledge derived from medical reports is often overlooked in prior research, which is important for enhancing the understanding of disease progression and treatment outcomes. To address these problems, we propose a knowledge integrated multi-task learning (KIMTL) network that performs tumor segmentation and pathological complete response (pCR) prediction concurrently. Clinical knowledge is merged with extracted high-level image features to enhance prediction performance. The attention mechanism effectively leverages the inter-channel and inter-spatial relationships within features, thereby enhancing network effectiveness. The proposed multi-task learning network optimizes the balance between segmentation and prediction tasks using uncertainty weight loss. The experimental results from a dataset of 216 cases indicate that KIMTL could improve the performance of both tasks, particularly the prediction task (AUC = 0.816). Specifically, in the prediction task, the AUC increases from 0.789 to 0.816. In the segmentation task, the Jaccard index is improved from 0.710 to 0.740. Our study suggests that incorporating clinical domain knowledge into deep learning modeling can augment the performance of breast tumor segmentation and pCR prediction. KIMTL achieves promising performance and outperforms its single-task learning counterparts.