Pub Date : 2024-03-06DOI: 10.1016/j.compmedimag.2024.102366
Shuai Lu , Weihang Zhang , Jia Guo , Hanruo Liu , Huiqi Li , Ningli Wang
Anomaly detection is an important yet challenging task in medical image analysis. Most anomaly detection methods are based on reconstruction, but the performance of reconstruction-based methods is limited due to over-reliance on pixel-level losses. To address the limitation, we propose a patch-wise contrastive learning-based auto-encoder for medical anomaly detection. The key contribution is the patch-wise contrastive learning loss that provides supervision on local semantics to enforce semantic consistency between corresponding input–output patches. Contrastive learning pulls corresponding patch pairs closer while pushing non-corresponding ones apart between input and output, enabling the model to learn local normal features better and improve discriminability on anomalous regions. Additionally, we design an anomaly score based on local semantic discrepancies to pinpoint abnormalities by comparing feature difference rather than pixel variations. Extensive experiments on three public datasets (i.e., brain MRI, retinal OCT, and chest X-ray) achieve state-of-the-art performance, with our method achieving over 99% AUC on retinal and brain images. Both the contrastive patch-wise supervision and patch-discrepancy score provide targeted advancements to overcome the weaknesses in existing approaches.
异常检测是医学图像分析中一项重要而又具有挑战性的任务。大多数异常检测方法都基于重构,但由于过度依赖像素级损失,基于重构的方法性能有限。为了解决这个问题,我们提出了一种用于医学异常检测的基于补丁对比学习的自动编码器。该方法的主要贡献在于补丁对比学习损失,它提供了对局部语义的监督,以加强相应输入输出补丁之间的语义一致性。对比学习将相应的补丁对拉近,同时将输入和输出之间不对应的补丁对拉开,从而使模型能够更好地学习局部正常特征,并提高对异常区域的判别能力。此外,我们还设计了基于局部语义差异的异常评分,通过比较特征差异而不是像素变化来精确定位异常。我们在三个公共数据集(即脑磁共振成像、视网膜 OCT 和胸部 X 光)上进行了广泛的实验,取得了最先进的性能,我们的方法在视网膜和脑图像上的 AUC 超过 99%。对比性斑块监督和斑块差异得分都为克服现有方法的弱点提供了有针对性的进步。
{"title":"PatchCL-AE: Anomaly detection for medical images using patch-wise contrastive learning-based auto-encoder","authors":"Shuai Lu , Weihang Zhang , Jia Guo , Hanruo Liu , Huiqi Li , Ningli Wang","doi":"10.1016/j.compmedimag.2024.102366","DOIUrl":"https://doi.org/10.1016/j.compmedimag.2024.102366","url":null,"abstract":"<div><p>Anomaly detection is an important yet challenging task in medical image analysis. Most anomaly detection methods are based on reconstruction, but the performance of reconstruction-based methods is limited due to over-reliance on pixel-level losses. To address the limitation, we propose a patch-wise contrastive learning-based auto-encoder for medical anomaly detection. The key contribution is the patch-wise contrastive learning loss that provides supervision on local semantics to enforce semantic consistency between corresponding input–output patches. Contrastive learning pulls corresponding patch pairs closer while pushing non-corresponding ones apart between input and output, enabling the model to learn local normal features better and improve discriminability on anomalous regions. Additionally, we design an anomaly score based on local semantic discrepancies to pinpoint abnormalities by comparing feature difference rather than pixel variations. Extensive experiments on three public datasets (<em>i.e.</em>, brain MRI, retinal OCT, and chest X-ray) achieve state-of-the-art performance, with our method achieving over 99% AUC on retinal and brain images. Both the contrastive patch-wise supervision and patch-discrepancy score provide targeted advancements to overcome the weaknesses in existing approaches.</p></div>","PeriodicalId":50631,"journal":{"name":"Computerized Medical Imaging and Graphics","volume":null,"pages":null},"PeriodicalIF":5.7,"publicationDate":"2024-03-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140095986","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-03-01DOI: 10.1016/j.compmedimag.2024.102364
Zimeng Tan , Jianjiang Feng , Wangsheng Lu , Yin Yin , Guangming Yang , Jie Zhou
Vascular landmark detection plays an important role in medical analysis and clinical treatment. However, due to the complex topology and similar local appearance around landmarks, the popular heatmap regression based methods always suffer from the landmark confusion problem. Vascular landmarks are connected by vascular segments and have special spatial correlations, which can be utilized for performance improvement. In this paper, we propose a multi-task global optimization-based framework for accurate and automatic vascular landmark detection. A multi-task deep learning network is exploited to accomplish landmark heatmap regression, vascular semantic segmentation, and orientation field regression simultaneously. The two auxiliary objectives are highly correlated with the heatmap regression task and help the network incorporate the structural prior knowledge. During inference, instead of performing a max-voting strategy, we propose a global optimization-based post-processing method for final landmark decision. The spatial relationships between neighboring landmarks are utilized explicitly to tackle the landmark confusion problem. We evaluated our method on a cerebral MRA dataset with 564 volumes, a cerebral CTA dataset with 510 volumes, and an aorta CTA dataset with 50 volumes. The experiments demonstrate that the proposed method is effective for vascular landmark localization and achieves state-of-the-art performance.
{"title":"Multi-task global optimization-based method for vascular landmark detection","authors":"Zimeng Tan , Jianjiang Feng , Wangsheng Lu , Yin Yin , Guangming Yang , Jie Zhou","doi":"10.1016/j.compmedimag.2024.102364","DOIUrl":"https://doi.org/10.1016/j.compmedimag.2024.102364","url":null,"abstract":"<div><p>Vascular landmark detection plays an important role in medical analysis and clinical treatment. However, due to the complex topology and similar local appearance around landmarks, the popular heatmap regression based methods always suffer from the landmark confusion problem. Vascular landmarks are connected by vascular segments and have special spatial correlations, which can be utilized for performance improvement. In this paper, we propose a multi-task global optimization-based framework for accurate and automatic vascular landmark detection. A multi-task deep learning network is exploited to accomplish landmark heatmap regression, vascular semantic segmentation, and orientation field regression simultaneously. The two auxiliary objectives are highly correlated with the heatmap regression task and help the network incorporate the structural prior knowledge. During inference, instead of performing a max-voting strategy, we propose a global optimization-based post-processing method for final landmark decision. The spatial relationships between neighboring landmarks are utilized explicitly to tackle the landmark confusion problem. We evaluated our method on a cerebral MRA dataset with 564 volumes, a cerebral CTA dataset with 510 volumes, and an aorta CTA dataset with 50 volumes. The experiments demonstrate that the proposed method is effective for vascular landmark localization and achieves state-of-the-art performance.</p></div>","PeriodicalId":50631,"journal":{"name":"Computerized Medical Imaging and Graphics","volume":null,"pages":null},"PeriodicalIF":5.7,"publicationDate":"2024-03-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140016089","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-03-01DOI: 10.1016/j.compmedimag.2024.102363
Tejas Sudharshan Mathai , Thomas C. Shen , Daniel C. Elton , Sungwon Lee , Zhiyong Lu , Ronald M. Summers
Reliable localization of lymph nodes (LNs) in multi-parametric MRI (mpMRI) studies plays a major role in the assessment of lymphadenopathy and staging of metastatic disease. Radiologists routinely measure the nodal size in order to distinguish benign from malignant nodes, which require subsequent cancer staging. However, identification of lymph nodes is a cumbersome task due to their myriad appearances in mpMRI studies. Multiple sequences are acquired in mpMRI studies, including T2 fat suppressed (T2FS) and diffusion weighted imaging (DWI) sequences among others; consequently, the sizing of LNs is rendered challenging due to the variety of signal intensities in these sequences. Furthermore, radiologists can miss potentially metastatic LNs during a busy clinical day. To lighten these imaging and workflow challenges, we propose a computer-aided detection (CAD) pipeline to detect both benign and malignant LNs in the body for their subsequent measurement. We employed the recently proposed Dynamic Head (DyHead) neural network to detect LNs in mpMRI studies that were acquired using a variety of scanners and exam protocols. The T2FS and DWI series were co-registered, and a selective augmentation technique called Intra-Label LISA (ILL) was used to blend the two volumes with the interpolation factor drawn from a Beta distribution. In this way, ILL diversified the samples that the model encountered during the training phase, while the requirement for both sequences to be present at test time was nullified. Our results showed a mean average precision (mAP) of 53.5% and a sensitivity of 78% with ILL at 4 FP/vol. This corresponded to an improvement of 10% in mAP and 12% in sensitivity at 4FP (p ¡ 0.05) respectively over current LN detection approaches evaluated on the same dataset. We also established the out-of-distribution robustness of the DyHead model by training it on data acquired by a Siemens Aera scanner and testing it on data from the Siemens Verio, Siemens Biograph mMR, and Philips Achieva scanners. Our pilot work represents an important first step towards automated detection, segmentation, and classification of lymph nodes in mpMRI.
{"title":"Detection of abdominopelvic lymph nodes in multi-parametric MRI","authors":"Tejas Sudharshan Mathai , Thomas C. Shen , Daniel C. Elton , Sungwon Lee , Zhiyong Lu , Ronald M. Summers","doi":"10.1016/j.compmedimag.2024.102363","DOIUrl":"https://doi.org/10.1016/j.compmedimag.2024.102363","url":null,"abstract":"<div><p>Reliable localization of lymph nodes (LNs) in multi-parametric MRI (mpMRI) studies plays a major role in the assessment of lymphadenopathy and staging of metastatic disease. Radiologists routinely measure the nodal size in order to distinguish benign from malignant nodes, which require subsequent cancer staging. However, identification of lymph nodes is a cumbersome task due to their myriad appearances in mpMRI studies. Multiple sequences are acquired in mpMRI studies, including T2 fat suppressed (T2FS) and diffusion weighted imaging (DWI) sequences among others; consequently, the sizing of LNs is rendered challenging due to the variety of signal intensities in these sequences. Furthermore, radiologists can miss potentially metastatic LNs during a busy clinical day. To lighten these imaging and workflow challenges, we propose a computer-aided detection (CAD) pipeline to detect both benign and malignant LNs in the body for their subsequent measurement. We employed the recently proposed Dynamic Head (DyHead) neural network to detect LNs in mpMRI studies that were acquired using a variety of scanners and exam protocols. The T2FS and DWI series were co-registered, and a selective augmentation technique called Intra-Label LISA (ILL) was used to blend the two volumes with the interpolation factor drawn from a Beta distribution. In this way, ILL diversified the samples that the model encountered during the training phase, while the requirement for both sequences to be present at test time was nullified. Our results showed a mean average precision (mAP) of 53.5% and a sensitivity of <span><math><mo>∼</mo></math></span>78% with ILL at 4 FP/vol. This corresponded to an improvement of <span><math><mo>≥</mo></math></span>10% in mAP and <span><math><mo>≥</mo></math></span>12% in sensitivity at 4FP (<em>p</em> ¡ 0.05) respectively over current LN detection approaches evaluated on the same dataset. We also established the out-of-distribution robustness of the DyHead model by training it on data acquired by a Siemens Aera scanner and testing it on data from the Siemens Verio, Siemens Biograph mMR, and Philips Achieva scanners. Our pilot work represents an important first step towards automated detection, segmentation, and classification of lymph nodes in mpMRI.</p></div>","PeriodicalId":50631,"journal":{"name":"Computerized Medical Imaging and Graphics","volume":null,"pages":null},"PeriodicalIF":5.7,"publicationDate":"2024-03-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140030312","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Automatic retinal arteriovenous classification can assist ophthalmologists in disease early diagnosis. Deep learning-based methods and topological graph-based methods have become the main solutions for retinal arteriovenous classification in recent years. This paper reviews the automatic retinal arteriovenous classification methods from 2003 to 2022. Firstly, we compare different methods and provide comparison tables of the summary results. Secondly, we complete the classification of the public arteriovenous classification datasets and provide the annotation development tables of different datasets. Finally, we sort out the challenges of evaluation methods and provide a comprehensive evaluation system. Quantitative and qualitative analysis shows the changes in research hotspots over time, Quantitative and qualitative analyses reveal the evolution of research hotspots over time, highlighting the significance of exploring the integration of deep learning with topological information in future research.
{"title":"Automatic artery/vein classification methods for retinal blood vessel: A review","authors":"Qihan Chen , Jianqing Peng , Shen Zhao , Wanquan Liu","doi":"10.1016/j.compmedimag.2024.102355","DOIUrl":"https://doi.org/10.1016/j.compmedimag.2024.102355","url":null,"abstract":"<div><p>Automatic retinal arteriovenous classification can assist ophthalmologists in disease early diagnosis. Deep learning-based methods and topological graph-based methods have become the main solutions for retinal arteriovenous classification in recent years. This paper reviews the automatic retinal arteriovenous classification methods from 2003 to 2022. Firstly, we compare different methods and provide comparison tables of the summary results. Secondly, we complete the classification of the public arteriovenous classification datasets and provide the annotation development tables of different datasets. Finally, we sort out the challenges of evaluation methods and provide a comprehensive evaluation system. Quantitative and qualitative analysis shows the changes in research hotspots over time, Quantitative and qualitative analyses reveal the evolution of research hotspots over time, highlighting the significance of exploring the integration of deep learning with topological information in future research.</p></div>","PeriodicalId":50631,"journal":{"name":"Computerized Medical Imaging and Graphics","volume":null,"pages":null},"PeriodicalIF":5.7,"publicationDate":"2024-02-16","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"139907922","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-02-15DOI: 10.1016/j.compmedimag.2024.102346
Mumu Aktar , Yiming Xiao , Ali K.Z. Tehrani , Donatella Tampieri , Hassan Rivaz , Marta Kersten-Oertel
This study conducts collateral evaluation from ischemic damage using a deep learning-based Siamese network, addressing the challenges associated with a small and imbalanced dataset. The collateral network provides an alternative oxygen and nutrient supply pathway in ischemic stroke cases, influencing treatment decisions. Research in this area focuses on automated collateral assessment using deep learning (DL) methods to expedite decision-making processes and enhance accuracy. Our study employed a 3D ResNet-based Siamese network, referred to as SCANED, to classify collaterals as good/intermediate or poor. Utilizing non-contrast computed tomography (NCCT) images, the network automates collateral identification and assessment by analyzing tissue degeneration around the ischemic site. Relevant features from the left/right hemispheres were extracted, and Euclidean Distance (ED) was employed for similarity measurement. Finally, dichotomized classification of good/intermediate or poor collateral is performed by SCANED using an optimal threshold derived from ROC analysis. SCANED provides a sensitivity of 0.88, a specificity of 0.63, and a weighted F1 score of 0.86 in the dichotomized classification.
{"title":"SCANED: Siamese collateral assessment network for evaluation of collaterals from ischemic damage","authors":"Mumu Aktar , Yiming Xiao , Ali K.Z. Tehrani , Donatella Tampieri , Hassan Rivaz , Marta Kersten-Oertel","doi":"10.1016/j.compmedimag.2024.102346","DOIUrl":"https://doi.org/10.1016/j.compmedimag.2024.102346","url":null,"abstract":"<div><p>This study conducts collateral evaluation from ischemic damage using a deep learning-based Siamese network, addressing the challenges associated with a small and imbalanced dataset. The collateral network provides an alternative oxygen and nutrient supply pathway in ischemic stroke cases, influencing treatment decisions. Research in this area focuses on automated collateral assessment using deep learning (DL) methods to expedite decision-making processes and enhance accuracy. Our study employed a 3D ResNet-based Siamese network, referred to as SCANED, to classify collaterals as good/intermediate or poor. Utilizing non-contrast computed tomography (NCCT) images, the network automates collateral identification and assessment by analyzing tissue degeneration around the ischemic site. Relevant features from the left/right hemispheres were extracted, and Euclidean Distance (ED) was employed for similarity measurement. Finally, dichotomized classification of good/intermediate or poor collateral is performed by SCANED using an optimal threshold derived from ROC analysis. SCANED provides a sensitivity of 0.88, a specificity of 0.63, and a weighted F1 score of 0.86 in the dichotomized classification.</p></div>","PeriodicalId":50631,"journal":{"name":"Computerized Medical Imaging and Graphics","volume":null,"pages":null},"PeriodicalIF":5.7,"publicationDate":"2024-02-15","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.sciencedirect.com/science/article/pii/S0895611124000235/pdfft?md5=6b0fbf08d111cd87a2314d489eeb3849&pid=1-s2.0-S0895611124000235-main.pdf","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"139738796","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-02-10DOI: 10.1016/j.compmedimag.2024.102353
Xinru Chen , Yao Zhao , Laurence E. Court , He Wang , Tinsu Pan , Jack Phan , Xin Wang , Yao Ding , Jinzhong Yang
Creating synthetic CT (sCT) from magnetic resonance (MR) images enables MR-based treatment planning in radiation therapy. However, the MR images used for MR-guided adaptive planning are often truncated in the boundary regions due to the limited field of view and the need for sequence optimization. Consequently, the sCT generated from these truncated MR images lacks complete anatomic information, leading to dose calculation error for MR-based adaptive planning. We propose a novel structure-completion generative adversarial network (SC-GAN) to generate sCT with full anatomic details from the truncated MR images. To enable anatomy compensation, we expand input channels of the CT generator by including a body mask and introduce a truncation loss between sCT and real CT. The body mask for each patient was automatically created from the simulation CT scans and transformed to daily MR images by rigid registration as another input for our SC-GAN in addition to the MR images. The truncation loss was constructed by implementing either an auto-segmentor or an edge detector to penalize the difference in body outlines between sCT and real CT. The experimental results show that our SC-GAN achieved much improved accuracy of sCT generation in both truncated and untruncated regions compared to the original cycleGAN and conditional GAN methods.
{"title":"SC-GAN: Structure-completion generative adversarial network for synthetic CT generation from MR images with truncated anatomy","authors":"Xinru Chen , Yao Zhao , Laurence E. Court , He Wang , Tinsu Pan , Jack Phan , Xin Wang , Yao Ding , Jinzhong Yang","doi":"10.1016/j.compmedimag.2024.102353","DOIUrl":"https://doi.org/10.1016/j.compmedimag.2024.102353","url":null,"abstract":"<div><p>Creating synthetic CT (sCT) from magnetic resonance (MR) images enables MR-based treatment planning in radiation therapy. However, the MR images used for MR-guided adaptive planning are often truncated in the boundary regions due to the limited field of view and the need for sequence optimization. Consequently, the sCT generated from these truncated MR images lacks complete anatomic information, leading to dose calculation error for MR-based adaptive planning. We propose a novel structure-completion generative adversarial network (SC-GAN) to generate sCT with full anatomic details from the truncated MR images. To enable anatomy compensation, we expand input channels of the CT generator by including a body mask and introduce a truncation loss between sCT and real CT. The body mask for each patient was automatically created from the simulation CT scans and transformed to daily MR images by rigid registration as another input for our SC-GAN in addition to the MR images. The truncation loss was constructed by implementing either an auto-segmentor or an edge detector to penalize the difference in body outlines between sCT and real CT. The experimental results show that our SC-GAN achieved much improved accuracy of sCT generation in both truncated and untruncated regions compared to the original cycleGAN and conditional GAN methods.</p></div>","PeriodicalId":50631,"journal":{"name":"Computerized Medical Imaging and Graphics","volume":null,"pages":null},"PeriodicalIF":5.7,"publicationDate":"2024-02-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"139915280","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-02-09DOI: 10.1016/j.compmedimag.2024.102356
Amine Sadikine , Bogdan Badic , Jean-Pierre Tasu , Vincent Noblet , Pascal Ballet , Dimitris Visvikis , Pierre-Henri Conze
The extraction of abdominal structures using deep learning has recently experienced a widespread interest in medical image analysis. Automatic abdominal organ and vessel segmentation is highly desirable to guide clinicians in computer-assisted diagnosis, therapy, or surgical planning. Despite a good ability to extract large organs, the capacity of U-Net inspired architectures to automatically delineate smaller structures remains a major issue, especially given the increase in receptive field size as we go deeper into the network. To deal with various abdominal structure sizes while exploiting efficient geometric constraints, we present a novel approach that integrates into deep segmentation shape priors from a semi-overcomplete convolutional auto-encoder (S-OCAE) embedding. Compared to standard convolutional auto-encoders (CAE), it exploits an over-complete branch that projects data onto higher dimensions to better characterize anatomical structures with a small spatial extent. Experiments on abdominal organs and vessel delineation performed on various publicly available datasets highlight the effectiveness of our method compared to state-of-the-art, including U-Net trained without and with shape priors from a traditional CAE. Exploiting a semi-overcomplete convolutional auto-encoder embedding as shape priors improves the ability of deep segmentation models to provide realistic and accurate abdominal structure contours.
{"title":"Improving abdominal image segmentation with overcomplete shape priors","authors":"Amine Sadikine , Bogdan Badic , Jean-Pierre Tasu , Vincent Noblet , Pascal Ballet , Dimitris Visvikis , Pierre-Henri Conze","doi":"10.1016/j.compmedimag.2024.102356","DOIUrl":"https://doi.org/10.1016/j.compmedimag.2024.102356","url":null,"abstract":"<div><p>The extraction of abdominal structures using deep learning has recently experienced a widespread interest in medical image analysis. Automatic abdominal organ and vessel segmentation is highly desirable to guide clinicians in computer-assisted diagnosis, therapy, or surgical planning. Despite a good ability to extract large organs, the capacity of U-Net inspired architectures to automatically delineate smaller structures remains a major issue, especially given the increase in receptive field size as we go deeper into the network. To deal with various abdominal structure sizes while exploiting efficient geometric constraints, we present a novel approach that integrates into deep segmentation shape priors from a semi-overcomplete convolutional auto-encoder (S-OCAE) embedding. Compared to standard convolutional auto-encoders (CAE), it exploits an over-complete branch that projects data onto higher dimensions to better characterize anatomical structures with a small spatial extent. Experiments on abdominal organs and vessel delineation performed on various publicly available datasets highlight the effectiveness of our method compared to state-of-the-art, including U-Net trained without and with shape priors from a traditional CAE. Exploiting a semi-overcomplete convolutional auto-encoder embedding as shape priors improves the ability of deep segmentation models to provide realistic and accurate abdominal structure contours.</p></div>","PeriodicalId":50631,"journal":{"name":"Computerized Medical Imaging and Graphics","volume":null,"pages":null},"PeriodicalIF":5.7,"publicationDate":"2024-02-09","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.sciencedirect.com/science/article/pii/S0895611124000338/pdfft?md5=e83e8e8de56fb1f5b580a68d5fed492b&pid=1-s2.0-S0895611124000338-main.pdf","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"139714471","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-02-09DOI: 10.1016/j.compmedimag.2024.102347
Yiqing Liu , Farhad R. Nezami , Elazer R. Edelman
Characterizing coronary calcified plaque (CCP) provides essential insight into diagnosis and treatment of atherosclerosis. Intravascular optical coherence tomography (OCT) offers significant advantages for detecting CCP and even automated segmentation with recent advances in deep learning techniques. Most of current methods have achieved promising results by adopting existing convolution neural networks (CNNs) in computer vision domain. However, their performance can be detrimentally affected by unseen plaque patterns and artifacts due to inherent limitation of CNNs in contextual reasoning. To overcome this obstacle, we proposed a Transformer-based pyramid network called AFS-TPNet for robust, end-to-end segmentation of CCP from OCT images. Its encoder is built upon CSWin Transformer architecture, allowing for better perceptual understanding of calcified arteries at a higher semantic level. Specifically, an augmented feature split (AFS) module and residual convolutional position encoding (RCPE) mechanism are designed to effectively enhance the capability of Transformer in capturing both fine-grained features and global contexts. Extensive experiments showed that AFS-TPNet trained using Lovasz Loss achieved superior performance in segmentation CCP under various contexts, surpassing prior state-of-the-art CNN and Transformer architectures by more than 6.58% intersection over union (IoU) score. The application of this promising method to extract CCP features is expected to enhance clinical intervention and translational research using OCT.
冠状动脉钙化斑块(CCP)的特征为动脉粥样硬化的诊断和治疗提供了重要依据。血管内光学相干断层扫描(OCT)在检测冠状动脉钙化斑块(CCP)方面具有显著优势,随着深度学习技术的最新进展,甚至可以实现自动分割。目前的大多数方法都采用了计算机视觉领域现有的卷积神经网络(CNN),取得了可喜的成果。然而,由于卷积神经网络在上下文推理方面的固有局限性,它们的性能可能会受到未见斑块模式和伪影的不利影响。为了克服这一障碍,我们提出了一种名为 AFS-TPNet 的基于变换器的金字塔网络,用于从 OCT 图像中对 CCP 进行稳健的端到端分割。它的编码器建立在 CSWin Transformer 架构之上,可以在更高的语义层面上更好地感知钙化动脉。具体来说,设计了增强特征分割(AFS)模块和残差卷积位置编码(RCPE)机制,以有效增强 Transformer 在捕捉细粒度特征和全局上下文方面的能力。广泛的实验表明,使用 Lovasz Loss 训练的 AFS-TPNet 在各种上下文条件下分割 CCP 时都取得了优异的性能,超过了之前最先进的 CNN 和 Transformer 架构 6.58% 以上的 intersection over union (IoU) 分数。应用这种前景广阔的方法来提取 CCP 特征,有望利用 OCT 加强临床干预和转化研究。
{"title":"A transformer-based pyramid network for coronary calcified plaque segmentation in intravascular optical coherence tomography images","authors":"Yiqing Liu , Farhad R. Nezami , Elazer R. Edelman","doi":"10.1016/j.compmedimag.2024.102347","DOIUrl":"10.1016/j.compmedimag.2024.102347","url":null,"abstract":"<div><p>Characterizing coronary calcified plaque (CCP) provides essential insight into diagnosis and treatment of atherosclerosis. Intravascular optical coherence tomography (OCT) offers significant advantages for detecting CCP and even automated segmentation with recent advances in deep learning techniques. Most of current methods have achieved promising results by adopting existing convolution neural networks (CNNs) in computer vision domain. However, their performance can be detrimentally affected by unseen plaque patterns and artifacts due to inherent limitation of CNNs in contextual reasoning. To overcome this obstacle, we proposed a Transformer-based pyramid network called AFS-TPNet for robust, end-to-end segmentation of CCP from OCT images. Its encoder is built upon CSWin Transformer architecture, allowing for better perceptual understanding of calcified arteries at a higher semantic level. Specifically, an augmented feature split (AFS) module and residual convolutional position encoding (RCPE) mechanism are designed to effectively enhance the capability of Transformer in capturing both fine-grained features and global contexts. Extensive experiments showed that AFS-TPNet trained using Lovasz Loss achieved superior performance in segmentation CCP under various contexts, surpassing prior state-of-the-art CNN and Transformer architectures by more than 6.58% intersection over union (IoU) score. The application of this promising method to extract CCP features is expected to enhance clinical intervention and translational research using OCT.</p></div>","PeriodicalId":50631,"journal":{"name":"Computerized Medical Imaging and Graphics","volume":null,"pages":null},"PeriodicalIF":5.7,"publicationDate":"2024-02-09","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"139718035","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-02-08DOI: 10.1016/j.compmedimag.2024.102350
Luís Serrador , Francesca Pia Villani , Sara Moccia , Cristina P. Santos
Recent advances in medical imaging have highlighted the critical development of algorithms for individual vertebral segmentation on computed tomography (CT) scans. Essential for diagnostic accuracy and treatment planning in orthopaedics, neurosurgery and oncology, these algorithms face challenges in clinical implementation, including integration into healthcare systems. Consequently, our focus lies in exploring the application of knowledge distillation (KD) methods to train shallower networks capable of efficiently segmenting vertebrae in CT scans. This approach aims to reduce segmentation time, enhance suitability for emergency cases, and optimize computational and memory resource efficiency. Building upon prior research in the field, a two-step segmentation approach was employed. Firstly, the spine’s location was determined by predicting a heatmap, indicating the probability of each voxel belonging to the spine. Subsequently, an iterative segmentation of vertebrae was performed from the top to the bottom of the CT volume over the located spine, using a memory instance to record the already segmented vertebrae. KD methods were implemented by training a teacher network with performance similar to that found in the literature, and this knowledge was distilled to a shallower network (student). Two KD methods were applied: (1) using the soft outputs of both networks and (2) matching logits. Two publicly available datasets, comprising 319 CT scans from 300 patients and a total of 611 cervical, 2387 thoracic, and 1507 lumbar vertebrae, were used. To ensure dataset balance and robustness, effective data augmentation methods were applied, including cleaning the memory instance to replicate the first vertebra segmentation. The teacher network achieved an average Dice similarity coefficient (DSC) of 88.22% and a Hausdorff distance (HD) of 7.71 mm, showcasing performance similar to other approaches in the literature. Through knowledge distillation from the teacher network, the student network’s performance improved, with an average DSC increasing from 75.78% to 84.70% and an HD decreasing from 15.17 mm to 8.08 mm. Compared to other methods, our teacher network exhibited up to 99.09% fewer parameters, 90.02% faster inference time, 88.46% shorter total segmentation time, and 89.36% less associated carbon (CO2) emission rate. Regarding our student network, it featured 75.00% fewer parameters than our teacher, resulting in a 36.15% reduction in inference time, a 33.33% decrease in total segmentation time, and a 42.96% reduction in CO2 emissions. This study marks the first exploration of applying KD to the problem of individual vertebrae segmentation in CT, demonstrating the feasibility of achieving comparable performance to existing methods using smaller neural networks.
{"title":"Knowledge distillation on individual vertebrae segmentation exploiting 3D U-Net","authors":"Luís Serrador , Francesca Pia Villani , Sara Moccia , Cristina P. Santos","doi":"10.1016/j.compmedimag.2024.102350","DOIUrl":"https://doi.org/10.1016/j.compmedimag.2024.102350","url":null,"abstract":"<div><p>Recent advances in medical imaging have highlighted the critical development of algorithms for individual vertebral segmentation on computed tomography (CT) scans. Essential for diagnostic accuracy and treatment planning in orthopaedics, neurosurgery and oncology, these algorithms face challenges in clinical implementation, including integration into healthcare systems. Consequently, our focus lies in exploring the application of knowledge distillation (KD) methods to train shallower networks capable of efficiently segmenting vertebrae in CT scans. This approach aims to reduce segmentation time, enhance suitability for emergency cases, and optimize computational and memory resource efficiency. Building upon prior research in the field, a two-step segmentation approach was employed. Firstly, the spine’s location was determined by predicting a heatmap, indicating the probability of each voxel belonging to the spine. Subsequently, an iterative segmentation of vertebrae was performed from the top to the bottom of the CT volume over the located spine, using a memory instance to record the already segmented vertebrae. KD methods were implemented by training a teacher network with performance similar to that found in the literature, and this knowledge was distilled to a shallower network (student). Two KD methods were applied: (1) using the soft outputs of both networks and (2) matching logits. Two publicly available datasets, comprising 319 CT scans from 300 patients and a total of 611 cervical, 2387 thoracic, and 1507 lumbar vertebrae, were used. To ensure dataset balance and robustness, effective data augmentation methods were applied, including cleaning the memory instance to replicate the first vertebra segmentation. The teacher network achieved an average Dice similarity coefficient (DSC) of 88.22% and a Hausdorff distance (HD) of 7.71 mm, showcasing performance similar to other approaches in the literature. Through knowledge distillation from the teacher network, the student network’s performance improved, with an average DSC increasing from 75.78% to 84.70% and an HD decreasing from 15.17 mm to 8.08 mm. Compared to other methods, our teacher network exhibited up to 99.09% fewer parameters, 90.02% faster inference time, 88.46% shorter total segmentation time, and 89.36% less associated carbon (CO<sub>2</sub>) emission rate. Regarding our student network, it featured 75.00% fewer parameters than our teacher, resulting in a 36.15% reduction in inference time, a 33.33% decrease in total segmentation time, and a 42.96% reduction in CO<sub>2</sub> emissions. This study marks the first exploration of applying KD to the problem of individual vertebrae segmentation in CT, demonstrating the feasibility of achieving comparable performance to existing methods using smaller neural networks.</p></div>","PeriodicalId":50631,"journal":{"name":"Computerized Medical Imaging and Graphics","volume":null,"pages":null},"PeriodicalIF":5.7,"publicationDate":"2024-02-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.sciencedirect.com/science/article/pii/S0895611124000272/pdfft?md5=5527b04bad0cd774436ca9f2fd764d59&pid=1-s2.0-S0895611124000272-main.pdf","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"139714472","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-02-08DOI: 10.1016/j.compmedimag.2024.102348
Kai Lønning , Matthan W.A. Caan , Marlies E. Nowee , Jan-Jakob Sonke
Recurrent inference machines (RIM), a deep learning model that learns an iterative scheme for reconstructing sparsely sampled MRI, has been shown able to perform well on accelerated 2D and 3D MRI scans, learn from small datasets and generalize well to unseen types of data. Here we propose the dynamic recurrent inference machine (DRIM) for reconstructing sparsely sampled 4D MRI by exploiting correlations between respiratory states. The DRIM was applied to a 4D protocol for MR-guided radiotherapy of liver lesions based on repetitive interleaved coronal 2D multi-slice -weighted acquisitions. We demonstrate with an ablation study that the DRIM outperforms the RIM, increasing the SSIM score from about 0.89 to 0.95. The DRIM allowed for an approximately 2.7 times faster scan time than the current clinical protocol with only a slight loss in image sharpness. Correlations between slice locations can also be used, but were found to be of less importance, as were a majority of tested variations in network architecture, as long as the respiratory states are processed by the network. Through cross-validation, the DRIM is also shown to be robust in terms of training data. We further demonstrate a good performance across a large range of subsampling factors, and conclude through an evaluation by a radiation oncologist that reconstructed images of the liver contour and inner structures are of a clinically acceptable standard at acceleration factors 10x and 8x, respectively. Finally, we show that binning the data with respect to respiratory states prior to reconstruction comes at a slight cost to reconstruction quality, but at greater speed of the overall protocol.
{"title":"Dynamic recurrent inference machines for accelerated MRI-guided radiotherapy of the liver","authors":"Kai Lønning , Matthan W.A. Caan , Marlies E. Nowee , Jan-Jakob Sonke","doi":"10.1016/j.compmedimag.2024.102348","DOIUrl":"https://doi.org/10.1016/j.compmedimag.2024.102348","url":null,"abstract":"<div><p>Recurrent inference machines (RIM), a deep learning model that learns an iterative scheme for reconstructing sparsely sampled MRI, has been shown able to perform well on accelerated 2D and 3D MRI scans, learn from small datasets and generalize well to unseen types of data. Here we propose the dynamic recurrent inference machine (DRIM) for reconstructing sparsely sampled 4D MRI by exploiting correlations between respiratory states. The DRIM was applied to a 4D protocol for MR-guided radiotherapy of liver lesions based on repetitive interleaved coronal 2D multi-slice <span><math><msub><mrow><mi>T</mi></mrow><mrow><mn>2</mn></mrow></msub></math></span>-weighted acquisitions. We demonstrate with an ablation study that the DRIM outperforms the RIM, increasing the SSIM score from about 0.89 to 0.95. The DRIM allowed for an approximately 2.7 times faster scan time than the current clinical protocol with only a slight loss in image sharpness. Correlations between slice locations can also be used, but were found to be of less importance, as were a majority of tested variations in network architecture, as long as the respiratory states are processed by the network. Through cross-validation, the DRIM is also shown to be robust in terms of training data. We further demonstrate a good performance across a large range of subsampling factors, and conclude through an evaluation by a radiation oncologist that reconstructed images of the liver contour and inner structures are of a clinically acceptable standard at acceleration factors 10x and 8x, respectively. Finally, we show that binning the data with respect to respiratory states prior to reconstruction comes at a slight cost to reconstruction quality, but at greater speed of the overall protocol.</p></div>","PeriodicalId":50631,"journal":{"name":"Computerized Medical Imaging and Graphics","volume":null,"pages":null},"PeriodicalIF":5.7,"publicationDate":"2024-02-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"139748249","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}