{"title":"用于腋窝淋巴结转移评估的多输入网络中的跨模式校准","authors":"Michela Gravina;Domiziana Santucci;Ermanno Cordelli;Paolo Soda;Carlo Sansone","doi":"10.1109/TAI.2024.3397246","DOIUrl":null,"url":null,"abstract":"The use of deep neural networks (DNNs) in medical images has enabled the development of solutions characterized by the need of leveraging information coming from multiple sources, raising the multimodal deep learning. DNNs are known for their ability to provide hierarchical and high-level representations of input data. This capability has led to the introduction of methods performing data fusion at an intermediate level, preserving the distinctiveness of the heterogeneous sources in modality-specific paths, while learning the way to define an effective combination in a shared representation. However, modeling the intricate relationships between different data remains an open issue. In this article, we aim to improve the integration of data coming from multiple sources. We introduce between layers belonging to different modality-specific paths a transfer module (TM) able to perform the cross-modality calibration of the extracted features, reducing the effects of the less discriminative ones. As case of study, we focus on the axillary lymph nodes (ALNs) metastasis evaluation in malignant breast cancer (BC), a crucial prognostic factor, affecting patient's survival. We propose a multi-input single-output 3-D convolutional neural network (CNN) that considers both images acquired with multiparametric magnetic resonance and clinical information. In particular, we assess the proposed methodology using four architectures, namely BasicNet and three ResNet variants, showing the improvement of the performance obtained by including the TM in the network configuration. Our results achieve up to 90% and 87% of accuracy and area under ROC curve, respectively when the ResNet10 is considered, surpassing various fusion strategies proposed in the literature.","PeriodicalId":73305,"journal":{"name":"IEEE transactions on artificial intelligence","volume":null,"pages":null},"PeriodicalIF":0.0000,"publicationDate":"2024-03-20","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10535471","citationCount":"0","resultStr":"{\"title\":\"Cross-Modality Calibration in Multi-Input Network for Axillary Lymph Node Metastasis Evaluation\",\"authors\":\"Michela Gravina;Domiziana Santucci;Ermanno Cordelli;Paolo Soda;Carlo Sansone\",\"doi\":\"10.1109/TAI.2024.3397246\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"The use of deep neural networks (DNNs) in medical images has enabled the development of solutions characterized by the need of leveraging information coming from multiple sources, raising the multimodal deep learning. DNNs are known for their ability to provide hierarchical and high-level representations of input data. This capability has led to the introduction of methods performing data fusion at an intermediate level, preserving the distinctiveness of the heterogeneous sources in modality-specific paths, while learning the way to define an effective combination in a shared representation. However, modeling the intricate relationships between different data remains an open issue. In this article, we aim to improve the integration of data coming from multiple sources. We introduce between layers belonging to different modality-specific paths a transfer module (TM) able to perform the cross-modality calibration of the extracted features, reducing the effects of the less discriminative ones. As case of study, we focus on the axillary lymph nodes (ALNs) metastasis evaluation in malignant breast cancer (BC), a crucial prognostic factor, affecting patient's survival. We propose a multi-input single-output 3-D convolutional neural network (CNN) that considers both images acquired with multiparametric magnetic resonance and clinical information. In particular, we assess the proposed methodology using four architectures, namely BasicNet and three ResNet variants, showing the improvement of the performance obtained by including the TM in the network configuration. Our results achieve up to 90% and 87% of accuracy and area under ROC curve, respectively when the ResNet10 is considered, surpassing various fusion strategies proposed in the literature.\",\"PeriodicalId\":73305,\"journal\":{\"name\":\"IEEE transactions on artificial intelligence\",\"volume\":null,\"pages\":null},\"PeriodicalIF\":0.0000,\"publicationDate\":\"2024-03-20\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10535471\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"IEEE transactions on artificial intelligence\",\"FirstCategoryId\":\"1085\",\"ListUrlMain\":\"https://ieeexplore.ieee.org/document/10535471/\",\"RegionNum\":0,\"RegionCategory\":null,\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"\",\"JCRName\":\"\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"IEEE transactions on artificial intelligence","FirstCategoryId":"1085","ListUrlMain":"https://ieeexplore.ieee.org/document/10535471/","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
Cross-Modality Calibration in Multi-Input Network for Axillary Lymph Node Metastasis Evaluation
The use of deep neural networks (DNNs) in medical images has enabled the development of solutions characterized by the need of leveraging information coming from multiple sources, raising the multimodal deep learning. DNNs are known for their ability to provide hierarchical and high-level representations of input data. This capability has led to the introduction of methods performing data fusion at an intermediate level, preserving the distinctiveness of the heterogeneous sources in modality-specific paths, while learning the way to define an effective combination in a shared representation. However, modeling the intricate relationships between different data remains an open issue. In this article, we aim to improve the integration of data coming from multiple sources. We introduce between layers belonging to different modality-specific paths a transfer module (TM) able to perform the cross-modality calibration of the extracted features, reducing the effects of the less discriminative ones. As case of study, we focus on the axillary lymph nodes (ALNs) metastasis evaluation in malignant breast cancer (BC), a crucial prognostic factor, affecting patient's survival. We propose a multi-input single-output 3-D convolutional neural network (CNN) that considers both images acquired with multiparametric magnetic resonance and clinical information. In particular, we assess the proposed methodology using four architectures, namely BasicNet and three ResNet variants, showing the improvement of the performance obtained by including the TM in the network configuration. Our results achieve up to 90% and 87% of accuracy and area under ROC curve, respectively when the ResNet10 is considered, surpassing various fusion strategies proposed in the literature.