Computerized Medical Imaging and Graphics最新文献_第5页

Dual attention model with reinforcement learning for classification of histology whole-slide images 利用强化学习的双重注意模型对组织学整张幻灯片图像进行分类。

IF 5.4 2区医学 Q1 ENGINEERING, BIOMEDICAL

Computerized Medical Imaging and Graphics

Pub Date : 2024-11-19 DOI: 10.1016/j.compmedimag.2024.102466

Manahil Raza , Ruqayya Awan , Raja Muhammad Saad Bashir , Talha Qaiser , Nasir M. Rajpoot

Digital whole slide images (WSIs) are generally captured at microscopic resolution and encompass extensive spatial data (several billions of pixels per image). Directly feeding these images to deep learning models is computationally intractable due to memory constraints, while downsampling the WSIs risks incurring information loss. Alternatively, splitting the WSIs into smaller patches (or tiles) may result in a loss of important contextual information. In this paper, we propose a novel dual attention approach, consisting of two main components, both inspired by the visual examination process of a pathologist: The first soft attention model processes a low magnification view of the WSI to identify relevant regions of interest (ROIs), followed by a custom sampling method to extract diverse and spatially distinct image tiles from the selected ROIs. The second component, the hard attention classification model further extracts a sequence of multi-resolution glimpses from each tile for classification. Since hard attention is non-differentiable, we train this component using reinforcement learning to predict the location of the glimpses. This approach allows the model to focus on essential regions instead of processing the entire tile, thereby aligning with a pathologist’s way of diagnosis. The two components are trained in an end-to-end fashion using a joint loss function to demonstrate the efficacy of the model. The proposed model was evaluated on two WSI-level classification problems: Human epidermal growth factor receptor 2 (HER2) scoring on breast cancer histology images and prediction of Intact/Loss status of two Mismatch Repair (MMR) biomarkers from colorectal cancer histology images. We show that the proposed model achieves performance better than or comparable to the state-of-the-art methods while processing less than 10% of the WSI at the highest magnification and reducing the time required to infer the WSI-level label by more than 75%. The code is available at github.

数字全切片图像（WSI）通常以显微镜分辨率捕获，包含大量空间数据（每幅图像数十亿像素）。由于内存限制，直接将这些图像输入深度学习模型在计算上难以实现，而对 WSIs 进行下采样则有可能导致信息丢失。另外，将 WSIs 分割成更小的斑块（或瓦片）可能会导致重要的上下文信息丢失。在本文中，我们提出了一种新颖的双重注意力方法，由两个主要部分组成，灵感均来自病理学家的视觉检查过程：第一个软注意力模型处理 WSI 的低倍视图，以识别相关的感兴趣区（ROI），然后采用自定义采样方法，从选定的 ROI 中提取不同的、空间上截然不同的图像块。第二个组件是硬注意力分类模型，它进一步从每个瓦片中提取多分辨率瞥视序列进行分类。由于硬注意力是无差别的，因此我们使用强化学习来预测瞥见的位置，从而对该组件进行训练。这种方法可以让模型专注于重要区域，而不是处理整个瓦片，从而与病理学家的诊断方法保持一致。使用联合损失函数对这两个组件进行端对端训练，以证明模型的有效性。我们在两个 WSI 级分类问题上对所提出的模型进行了评估：乳腺癌组织学图像上的人类表皮生长因子受体 2（HER2）评分，以及从结直肠癌组织学图像中预测两个错配修复（MMR）生物标记物的完好/丢失状态。我们的研究表明，所提出的模型性能优于或可与最先进的方法相媲美，同时在最高放大倍率下处理的 WSI 不到 10%，并将推断 WSI 级标签所需的时间减少了 75% 以上。代码可在 github 上获取。

{"title":"Dual attention model with reinforcement learning for classification of histology whole-slide images","authors":"Manahil Raza , Ruqayya Awan , Raja Muhammad Saad Bashir , Talha Qaiser , Nasir M. Rajpoot","doi":"10.1016/j.compmedimag.2024.102466","DOIUrl":"10.1016/j.compmedimag.2024.102466","url":null,"abstract":"<div><div>Digital whole slide images (WSIs) are generally captured at microscopic resolution and encompass extensive spatial data (several billions of pixels per image). Directly feeding these images to deep learning models is computationally intractable due to memory constraints, while downsampling the WSIs risks incurring information loss. Alternatively, splitting the WSIs into smaller patches (or tiles) may result in a loss of important contextual information. In this paper, we propose a novel dual attention approach, consisting of two main components, both inspired by the visual examination process of a pathologist: The first <em>soft</em> attention model processes a low magnification view of the WSI to identify relevant regions of interest (ROIs), followed by a custom sampling method to extract diverse and spatially distinct image tiles from the selected ROIs. The second component, the <em>hard</em> attention classification model further extracts a sequence of multi-resolution glimpses from each tile for classification. Since hard attention is non-differentiable, we train this component using reinforcement learning to predict the location of the glimpses. This approach allows the model to focus on essential regions instead of processing the entire tile, thereby aligning with a pathologist’s way of diagnosis. The two components are trained in an end-to-end fashion using a joint loss function to demonstrate the efficacy of the model. The proposed model was evaluated on two WSI-level classification problems: Human epidermal growth factor receptor 2 (HER2) scoring on breast cancer histology images and prediction of Intact/Loss status of two Mismatch Repair (MMR) biomarkers from colorectal cancer histology images. We show that the proposed model achieves performance better than or comparable to the state-of-the-art methods while processing less than 10% of the WSI at the highest magnification and reducing the time required to infer the WSI-level label by more than 75%. The code is available at <span><span>github</span><svg><path></path></svg></span>.</div></div>","PeriodicalId":50631,"journal":{"name":"Computerized Medical Imaging and Graphics","volume":"118 ","pages":"Article 102466"},"PeriodicalIF":5.4,"publicationDate":"2024-11-19","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142695994","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Circumpapillary OCT-based multi-sector analysis of retinal layer thickness in patients with glaucoma and high myopia 基于环状毛细血管 OCT 的青光眼和高度近视患者视网膜层厚度多扇区分析。

IF 5.4 2区医学 Q1 ENGINEERING, BIOMEDICAL

Computerized Medical Imaging and Graphics

Pub Date : 2024-11-19 DOI: 10.1016/j.compmedimag.2024.102464

Mateo Gende , Joaquim de Moura , Patricia Robles , Jose Fernández-Vigo , José M. Martínez-de-la-Casa , GlaucoClub AI, Julián García-Feijóo , Jorge Novo , Marcos Ortega

Glaucoma is the leading cause of irreversible blindness worldwide. The diagnosis process for glaucoma involves the measurement of the thickness of retinal layers in order to track its degeneration. The elongated shape of highly myopic eyes can hinder this diagnosis process, since it affects the OCT scanning process, producing deformations that can mimic or mask the degeneration caused by glaucoma. In this work, we present the first comprehensive cross-disease analysis that is focused on the anatomical structures most impacted in glaucoma and high myopia patients, facilitating precise differential diagnosis from those solely afflicted by myopia. To achieve this, a fully automatic approach for the retinal layer segmentation was specifically tailored for the accurate measurement of retinal thickness in both highly myopic and emmetropic eyes. To the best of our knowledge, this is the first approach proposed for the analysis of retinal layers in circumpapillary optical coherence tomography images that takes into account the elongation of the eyes in myopia, thus addressing critical diagnostic needs. The results from this study indicate that the temporal superior (mean difference

11.1 μ m

,

p < 0.05

), nasal inferior (

13.1 μ m

,

p < 0.01

) and temporal inferior (

13.3 μ m

,

p < 0.01

) sectors of the retinal nerve fibre layer show the most significant reduction in retinal thickness in patients of glaucoma and myopia with regards to patients of myopia.

青光眼是导致全球不可逆失明的主要原因。青光眼的诊断过程包括测量视网膜层的厚度，以跟踪其退化情况。高度近视眼的细长形状会影响 OCT 扫描过程，产生变形，从而模仿或掩盖青光眼引起的变性，因此会阻碍诊断过程。在这项工作中，我们首次提出了全面的跨疾病分析，重点关注青光眼和高度近视患者中受影响最大的解剖结构，从而有助于与单纯近视患者进行精确的鉴别诊断。为此，我们专门定制了一种全自动视网膜层分割方法，用于精确测量高度近视眼和弱视眼的视网膜厚度。据我们所知，这是第一种用于分析环状毛细血管光学相干断层扫描图像中视网膜层的方法，它考虑到了近视眼的眼球拉长问题，从而满足了重要的诊断需求。这项研究的结果表明，颞上部（平均差 11.1μm，p

{"title":"Circumpapillary OCT-based multi-sector analysis of retinal layer thickness in patients with glaucoma and high myopia","authors":"Mateo Gende , Joaquim de Moura , Patricia Robles , Jose Fernández-Vigo , José M. Martínez-de-la-Casa , GlaucoClub AI, Julián García-Feijóo , Jorge Novo , Marcos Ortega","doi":"10.1016/j.compmedimag.2024.102464","DOIUrl":"10.1016/j.compmedimag.2024.102464","url":null,"abstract":"<div><div>Glaucoma is the leading cause of irreversible blindness worldwide. The diagnosis process for glaucoma involves the measurement of the thickness of retinal layers in order to track its degeneration. The elongated shape of highly myopic eyes can hinder this diagnosis process, since it affects the OCT scanning process, producing deformations that can mimic or mask the degeneration caused by glaucoma. In this work, we present the first comprehensive cross-disease analysis that is focused on the anatomical structures most impacted in glaucoma and high myopia patients, facilitating precise differential diagnosis from those solely afflicted by myopia. To achieve this, a fully automatic approach for the retinal layer segmentation was specifically tailored for the accurate measurement of retinal thickness in both highly myopic and emmetropic eyes. To the best of our knowledge, this is the first approach proposed for the analysis of retinal layers in circumpapillary optical coherence tomography images that takes into account the elongation of the eyes in myopia, thus addressing critical diagnostic needs. The results from this study indicate that the temporal superior (mean difference <span><math><mrow><mn>11</mn><mo>.</mo><mn>1</mn><mspace></mspace><mi>μ</mi><mi>m</mi></mrow></math></span>, <span><math><mrow><mi>p</mi><mo><</mo><mn>0</mn><mo>.</mo><mn>05</mn></mrow></math></span>), nasal inferior (<span><math><mrow><mn>13</mn><mo>.</mo><mn>1</mn><mspace></mspace><mi>μ</mi><mi>m</mi></mrow></math></span>, <span><math><mrow><mi>p</mi><mo><</mo><mn>0</mn><mo>.</mo><mn>01</mn></mrow></math></span>) and temporal inferior (<span><math><mrow><mn>13</mn><mo>.</mo><mn>3</mn><mspace></mspace><mi>μ</mi><mi>m</mi></mrow></math></span>, <span><math><mrow><mi>p</mi><mo><</mo><mn>0</mn><mo>.</mo><mn>01</mn></mrow></math></span>) sectors of the retinal nerve fibre layer show the most significant reduction in retinal thickness in patients of glaucoma and myopia with regards to patients of myopia.</div></div>","PeriodicalId":50631,"journal":{"name":"Computerized Medical Imaging and Graphics","volume":"118 ","pages":"Article 102464"},"PeriodicalIF":5.4,"publicationDate":"2024-11-19","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142693815","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

CIS-UNet: Multi-class segmentation of the aorta in computed tomography angiography via context-aware shifted window self-attention CIS-UNet：通过上下文感知移动窗口自我关注，在计算机断层扫描血管造影中对主动脉进行多类分割。

IF 5.4 2区医学 Q1 ENGINEERING, BIOMEDICAL

Computerized Medical Imaging and Graphics

Pub Date : 2024-11-19 DOI: 10.1016/j.compmedimag.2024.102470

Muhammad Imran , Jonathan R. Krebs , Veera Rajasekhar Reddy Gopu , Brian Fazzone , Vishal Balaji Sivaraman , Amarjeet Kumar , Chelsea Viscardi , Robert Evans Heithaus , Benjamin Shickel , Yuyin Zhou , Michol A. Cooper , Wei Shao

Advancements in medical imaging and endovascular grafting have facilitated minimally invasive treatments for aortic diseases. Accurate 3D segmentation of the aorta and its branches is crucial for interventions, as inaccurate segmentation can lead to erroneous surgical planning and endograft construction. Previous methods simplified aortic segmentation as a binary image segmentation problem, overlooking the necessity of distinguishing between individual aortic branches. In this paper, we introduce Context-Infused Swin-UNet (CIS-UNet), a deep learning model designed for multi-class segmentation of the aorta and thirteen aortic branches. Combining the strengths of Convolutional Neural Networks (CNNs) and Swin transformers, CIS-UNet adopts a hierarchical encoder–decoder structure comprising a CNN encoder, a symmetric decoder, skip connections, and a novel Context-aware Shifted Window Self-Attention (CSW-SA) module as the bottleneck block. Notably, CSW-SA introduces a unique adaptation of the patch merging layer, distinct from its traditional use in the Swin transformers. CSW-SA efficiently condenses the feature map, providing a global spatial context, and enhances performance when applied at the bottleneck layer, offering superior computational efficiency and segmentation accuracy compared to the Swin transformers. We evaluated our model on computed tomography (CT) scans from 59 patients through a 4-fold cross-validation. CIS-UNet outperformed the state-of-the-art Swin UNetR segmentation model by achieving a superior mean Dice coefficient of 0.732 compared to 0.717 and a mean surface distance of 2.40 mm compared to 2.75 mm. CIS-UNet’s superior 3D aortic segmentation offers improved accuracy and optimization for planning endovascular treatments. Our dataset and code will be made publicly available at https://github.com/mirthAI/CIS-UNet.

医学成像和血管内移植技术的进步促进了主动脉疾病的微创治疗。对主动脉及其分支进行准确的三维分割对介入治疗至关重要，因为不准确的分割会导致错误的手术规划和内移植术。以前的方法将主动脉分割简化为二元图像分割问题，忽略了区分各个主动脉分支的必要性。本文介绍了一种深度学习模型 Context-Infused Swin-UNet（CIS-UNet），该模型专为主动脉和十三个主动脉分支的多类分割而设计。CIS-UNet 结合了卷积神经网络（CNN）和 Swin 变换器的优势，采用分层编码器-解码器结构，包括 CNN 编码器、对称解码器、跳转连接和作为瓶颈块的新型上下文感知移窗自注意（CSW-SA）模块。值得注意的是，CSW-SA 引入了对补丁合并层的独特调整，有别于斯温变换器中对其的传统使用。CSW-SA 能有效浓缩特征图，提供全局空间背景，并在瓶颈层应用时提高性能，与 Swin 变换器相比，具有更高的计算效率和分割精度。通过 4 倍交叉验证，我们对 59 名患者的计算机断层扫描（CT）扫描结果进行了评估。CIS-UNet 的平均 Dice 系数为 0.732，优于 0.717；平均表面距离为 2.40 毫米，优于 2.75 毫米。CIS-UNet 卓越的三维主动脉分割为规划血管内治疗提供了更高的准确性和优化性。我们的数据集和代码将在 https://github.com/mirthAI/CIS-UNet 上公布。

{"title":"CIS-UNet: Multi-class segmentation of the aorta in computed tomography angiography via context-aware shifted window self-attention","authors":"Muhammad Imran , Jonathan R. Krebs , Veera Rajasekhar Reddy Gopu , Brian Fazzone , Vishal Balaji Sivaraman , Amarjeet Kumar , Chelsea Viscardi , Robert Evans Heithaus , Benjamin Shickel , Yuyin Zhou , Michol A. Cooper , Wei Shao","doi":"10.1016/j.compmedimag.2024.102470","DOIUrl":"10.1016/j.compmedimag.2024.102470","url":null,"abstract":"<div><div>Advancements in medical imaging and endovascular grafting have facilitated minimally invasive treatments for aortic diseases. Accurate 3D segmentation of the aorta and its branches is crucial for interventions, as inaccurate segmentation can lead to erroneous surgical planning and endograft construction. Previous methods simplified aortic segmentation as a binary image segmentation problem, overlooking the necessity of distinguishing between individual aortic branches. In this paper, we introduce Context-Infused Swin-UNet (CIS-UNet), a deep learning model designed for multi-class segmentation of the aorta and thirteen aortic branches. Combining the strengths of Convolutional Neural Networks (CNNs) and Swin transformers, CIS-UNet adopts a hierarchical encoder–decoder structure comprising a CNN encoder, a symmetric decoder, skip connections, and a novel Context-aware Shifted Window Self-Attention (CSW-SA) module as the bottleneck block. Notably, CSW-SA introduces a unique adaptation of the patch merging layer, distinct from its traditional use in the Swin transformers. CSW-SA efficiently condenses the feature map, providing a global spatial context, and enhances performance when applied at the bottleneck layer, offering superior computational efficiency and segmentation accuracy compared to the Swin transformers. We evaluated our model on computed tomography (CT) scans from 59 patients through a 4-fold cross-validation. CIS-UNet outperformed the state-of-the-art Swin UNetR segmentation model by achieving a superior mean Dice coefficient of 0.732 compared to 0.717 and a mean surface distance of 2.40 mm compared to 2.75 mm. CIS-UNet’s superior 3D aortic segmentation offers improved accuracy and optimization for planning endovascular treatments. Our dataset and code will be made publicly available at <span><span>https://github.com/mirthAI/CIS-UNet</span><svg><path></path></svg></span>.</div></div>","PeriodicalId":50631,"journal":{"name":"Computerized Medical Imaging and Graphics","volume":"118 ","pages":"Article 102470"},"PeriodicalIF":5.4,"publicationDate":"2024-11-19","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142696048","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Cervical OCT image classification using contrastive masked autoencoders with Swin Transformer 使用带有 Swin 变换器的对比遮蔽自动编码器进行颈椎 OCT 图像分类。

IF 5.4 2区医学 Q1 ENGINEERING, BIOMEDICAL

Computerized Medical Imaging and Graphics

Pub Date : 2024-11-19 DOI: 10.1016/j.compmedimag.2024.102469

Qingbin Wang , Yuxuan Xiong , Hanfeng Zhu , Xuefeng Mu , Yan Zhang , Yutao Ma

Background and Objective:

Cervical cancer poses a major health threat to women globally. Optical coherence tomography (OCT) imaging has recently shown promise for non-invasive cervical lesion diagnosis. However, obtaining high-quality labeled cervical OCT images is challenging and time-consuming as they must correspond precisely with pathological results. The scarcity of such high-quality labeled data hinders the application of supervised deep-learning models in practical clinical settings. This study addresses the above challenge by proposing CMSwin, a novel self-supervised learning (SSL) framework combining masked image modeling (MIM) with contrastive learning based on the Swin-Transformer architecture to utilize abundant unlabeled cervical OCT images.

Methods:

In this contrastive-MIM framework, mixed image encoding is combined with a latent contextual regressor to solve the inconsistency problem between pre-training and fine-tuning and separate the encoder’s feature extraction task from the decoder’s reconstruction task, allowing the encoder to extract better image representations. Besides, contrastive losses at the patch and image levels are elaborately designed to leverage massive unlabeled data.

Results:

We validated the superiority of CMSwin over the state-of-the-art SSL approaches with five-fold cross-validation on an OCT image dataset containing 1,452 patients from a multi-center clinical study in China, plus two external validation sets from top-ranked Chinese hospitals: the Huaxi dataset from the West China Hospital of Sichuan University and the Xiangya dataset from the Xiangya Second Hospital of Central South University. A human-machine comparison experiment on the Huaxi and Xiangya datasets for volume-level binary classification also indicates that CMSwin can match or exceed the average level of four skilled medical experts, especially in identifying high-risk cervical lesions.

Conclusion:

Our work has great potential to assist gynecologists in intelligently interpreting cervical OCT images in clinical settings. Additionally, the integrated GradCAM module of CMSwin enables cervical lesion visualization and interpretation, providing good interpretability for gynecologists to diagnose cervical diseases efficiently.

背景和目的：宫颈癌对全球妇女的健康构成重大威胁。最近，光学相干断层扫描（OCT）成像技术在无创宫颈病变诊断方面显示出了良好的前景。然而，获得高质量的标记宫颈 OCT 图像具有挑战性且耗时，因为这些图像必须与病理结果精确对应。这种高质量标记数据的缺乏阻碍了有监督深度学习模型在实际临床环境中的应用。为了应对上述挑战，本研究提出了一种新颖的自我监督学习（SSL）框架--CMSwin，该框架结合了掩蔽图像建模（MIM）和基于 Swin-Transformer 架构的对比学习，以利用丰富的未标记颈椎 OCT 图像：在该对比-MIM 框架中，混合图像编码与潜在上下文回归器相结合，解决了预训练与微调之间的不一致问题，并将编码器的特征提取任务与解码器的重建任务分开，从而使编码器能够提取更好的图像表征。此外，我们还精心设计了补丁和图像层面的对比损失，以充分利用大量无标记数据：我们在一个包含来自中国多中心临床研究的 1,452 名患者的 OCT 图像数据集，以及两个来自中国顶级医院的外部验证集（四川大学华西医院的华西数据集和中南大学湘雅二医院的湘雅数据集）上进行了五倍交叉验证，验证了 CMSwin 优于最先进的 SSL 方法。在华西和湘雅数据集上进行的体积级二元分类的人机对比实验也表明，CMSwin 可以达到或超过四位技术熟练的医学专家的平均水平，尤其是在识别高危宫颈病变方面：我们的工作在帮助妇科医生在临床环境中智能解读宫颈 OCT 图像方面具有巨大潜力。此外，CMSwin 集成的 GradCAM 模块可实现宫颈病变的可视化和解读，为妇科医生有效诊断宫颈疾病提供了良好的可解读性。

{"title":"Cervical OCT image classification using contrastive masked autoencoders with Swin Transformer","authors":"Qingbin Wang , Yuxuan Xiong , Hanfeng Zhu , Xuefeng Mu , Yan Zhang , Yutao Ma","doi":"10.1016/j.compmedimag.2024.102469","DOIUrl":"10.1016/j.compmedimag.2024.102469","url":null,"abstract":"<div><h3>Background and Objective:</h3><div>Cervical cancer poses a major health threat to women globally. Optical coherence tomography (OCT) imaging has recently shown promise for non-invasive cervical lesion diagnosis. However, obtaining high-quality labeled cervical OCT images is challenging and time-consuming as they must correspond precisely with pathological results. The scarcity of such high-quality labeled data hinders the application of supervised deep-learning models in practical clinical settings. This study addresses the above challenge by proposing CMSwin, a novel self-supervised learning (SSL) framework combining masked image modeling (MIM) with contrastive learning based on the Swin-Transformer architecture to utilize abundant unlabeled cervical OCT images.</div></div><div><h3>Methods:</h3><div>In this contrastive-MIM framework, mixed image encoding is combined with a latent contextual regressor to solve the inconsistency problem between pre-training and fine-tuning and separate the encoder’s feature extraction task from the decoder’s reconstruction task, allowing the encoder to extract better image representations. Besides, contrastive losses at the patch and image levels are elaborately designed to leverage massive unlabeled data.</div></div><div><h3>Results:</h3><div>We validated the superiority of CMSwin over the state-of-the-art SSL approaches with five-fold cross-validation on an OCT image dataset containing 1,452 patients from a multi-center clinical study in China, plus two external validation sets from top-ranked Chinese hospitals: the Huaxi dataset from the West China Hospital of Sichuan University and the Xiangya dataset from the Xiangya Second Hospital of Central South University. A human-machine comparison experiment on the Huaxi and Xiangya datasets for volume-level binary classification also indicates that CMSwin can match or exceed the average level of four skilled medical experts, especially in identifying high-risk cervical lesions.</div></div><div><h3>Conclusion:</h3><div>Our work has great potential to assist gynecologists in intelligently interpreting cervical OCT images in clinical settings. Additionally, the integrated GradCAM module of CMSwin enables cervical lesion visualization and interpretation, providing good interpretability for gynecologists to diagnose cervical diseases efficiently.</div></div>","PeriodicalId":50631,"journal":{"name":"Computerized Medical Imaging and Graphics","volume":"118 ","pages":"Article 102469"},"PeriodicalIF":5.4,"publicationDate":"2024-11-19","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142693803","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

A Parkinson’s disease-related nuclei segmentation network based on CNN-Transformer interleaved encoder with feature fusion 基于 CNN-Transformer 交错编码器与特征融合的帕金森病相关核团分割网络

IF 5.4 2区医学 Q1 ENGINEERING, BIOMEDICAL

Computerized Medical Imaging and Graphics

Pub Date : 2024-11-19 DOI: 10.1016/j.compmedimag.2024.102465

Hongyi Chen , Junyan Fu , Xiao Liu , Zhiji Zheng , Xiao Luo , Kun Zhou , Zhijian Xu , Daoying Geng

Automatic segmentation of Parkinson’s disease (PD) related deep gray matter (DGM) nuclei based on brain magnetic resonance imaging (MRI) is significant in assisting the diagnosis of PD. However, due to the degenerative-induced changes in appearance, low tissue contrast, and tiny DGM nuclei size in elders’ brain MRI images, many existing segmentation models are limited in the application. To address these challenges, this paper proposes a PD-related DGM nuclei segmentation network to provide precise prior knowledge for aiding diagnosis PD. The encoder of network is designed as an alternating encoding structure where the convolutional neural network (CNN) captures spatial and depth texture features, while the Transformer complements global position information between DGM nuclei. Moreover, we propose a cascaded channel-spatial-wise block to fuse features extracted by the CNN and Transformer, thereby achieving more precise DGM nuclei segmentation. The decoder incorporates a symmetrical boundary attention module, leveraging the symmetrical structures of bilateral nuclei regions by constructing signed distance maps for symmetric differences, which optimizes segmentation boundaries. Furthermore, we employ a dynamic adaptive region of interests weighted Dice loss to enhance sensitivity towards smaller structures, thereby improving segmentation accuracy. In qualitative analysis, our method achieved optimal average values for PD-related DGM nuclei (DSC: 0.854, IOU: 0.750, HD95: 1.691 mm, ASD: 0.195 mm). Experiments conducted on multi-center clinical datasets and public datasets demonstrate the good generalizability of the proposed method. Furthermore, a volumetric analysis of segmentation results reveals significant differences between HCs and PDs. Our method holds promise for assisting clinicians in the rapid and accurate diagnosis of PD, offering a practical method for the imaging analysis of neurodegenerative diseases.

基于脑磁共振成像（MRI）的帕金森病（PD）相关深部灰质（DGM）核的自动分割对帕金森病的诊断具有重要意义。然而，由于退化引起的外观变化、低组织对比度以及老年人脑磁共振成像图像中 DGM 核的微小尺寸，许多现有的分割模型在应用中受到限制。针对这些挑战，本文提出了一种与帕金森病相关的 DGM 核分割网络，为帕金森病的辅助诊断提供精确的先验知识。该网络的编码器设计为交替编码结构，其中卷积神经网络（CNN）捕捉空间和深度纹理特征，而变换器则补充 DGM 核之间的全局位置信息。此外，我们还提出了一个级联通道空间块，以融合 CNN 和变换器提取的特征，从而实现更精确的 DGM 核分割。解码器集成了对称边界关注模块，通过构建对称差异的带符号距离图来利用双侧核区的对称结构，从而优化分割边界。此外，我们还采用了动态自适应兴趣区域加权骰子损失，以增强对较小结构的敏感性，从而提高分割准确性。在定性分析中，我们的方法获得了与帕金森病相关的 DGM 核的最佳平均值（DSC：0.854；IOU：0.750；HD95：1.691 毫米；ASD：0.195 毫米）。在多中心临床数据集和公共数据集上进行的实验证明，所提出的方法具有良好的普适性。此外，对分割结果进行的容积分析表明，HCs 和 PDs 之间存在显著差异。我们的方法有望帮助临床医生快速、准确地诊断帕金森病，为神经退行性疾病的成像分析提供了一种实用的方法。

{"title":"A Parkinson’s disease-related nuclei segmentation network based on CNN-Transformer interleaved encoder with feature fusion","authors":"Hongyi Chen , Junyan Fu , Xiao Liu , Zhiji Zheng , Xiao Luo , Kun Zhou , Zhijian Xu , Daoying Geng","doi":"10.1016/j.compmedimag.2024.102465","DOIUrl":"10.1016/j.compmedimag.2024.102465","url":null,"abstract":"<div><div>Automatic segmentation of Parkinson’s disease (PD) related deep gray matter (DGM) nuclei based on brain magnetic resonance imaging (MRI) is significant in assisting the diagnosis of PD. However, due to the degenerative-induced changes in appearance, low tissue contrast, and tiny DGM nuclei size in elders’ brain MRI images, many existing segmentation models are limited in the application. To address these challenges, this paper proposes a PD-related DGM nuclei segmentation network to provide precise prior knowledge for aiding diagnosis PD. The encoder of network is designed as an alternating encoding structure where the convolutional neural network (CNN) captures spatial and depth texture features, while the Transformer complements global position information between DGM nuclei. Moreover, we propose a cascaded channel-spatial-wise block to fuse features extracted by the CNN and Transformer, thereby achieving more precise DGM nuclei segmentation. The decoder incorporates a symmetrical boundary attention module, leveraging the symmetrical structures of bilateral nuclei regions by constructing signed distance maps for symmetric differences, which optimizes segmentation boundaries. Furthermore, we employ a dynamic adaptive region of interests weighted Dice loss to enhance sensitivity towards smaller structures, thereby improving segmentation accuracy. In qualitative analysis, our method achieved optimal average values for PD-related DGM nuclei (DSC: 0.854, IOU: 0.750, HD95: 1.691 mm, ASD: 0.195 mm). Experiments conducted on multi-center clinical datasets and public datasets demonstrate the good generalizability of the proposed method. Furthermore, a volumetric analysis of segmentation results reveals significant differences between HCs and PDs. Our method holds promise for assisting clinicians in the rapid and accurate diagnosis of PD, offering a practical method for the imaging analysis of neurodegenerative diseases.</div></div>","PeriodicalId":50631,"journal":{"name":"Computerized Medical Imaging and Graphics","volume":"118 ","pages":"Article 102465"},"PeriodicalIF":5.4,"publicationDate":"2024-11-19","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142700984","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Retinal structure guidance-and-adaption network for early Parkinson’s disease recognition based on OCT images 基于 OCT 图像的视网膜结构引导和适应网络，用于早期帕金森病识别

IF 5.4 2区医学 Q1 ENGINEERING, BIOMEDICAL

Computerized Medical Imaging and Graphics

Pub Date : 2024-11-19 DOI: 10.1016/j.compmedimag.2024.102463

Hanfeng Shi , Jiaqi Wei , Richu Jin , Jiaxin Peng , Xingyue Wang , Yan Hu , Xiaoqing Zhang , Jiang Liu

Parkinson’s disease (PD) is a leading neurodegenerative disease globally. Precise and objective PD diagnosis is significant for early intervention and treatment. Recent studies have shown significant correlations between retinal structure information and PD based on optical coherence tomography (OCT) images, providing another potential means for early PD recognition. However, how to exploit the retinal structure information (e.g., thickness and mean intensity) from different retinal layers to improve PD recognition performance has not been studied before. Motivated by the above observations, we first propose a structural prior knowledge extraction (SPKE) module to obtain the retinal structure feature maps; then, we develop a structure-guided-and-adaption attention (SGDA) module to fully leverage the potential of different retinal layers based on the extracted retinal structure feature maps. By embedding SPKE and SGDA modules at the low stage of deep neural networks (DNNs), a retinal structure-guided-and-adaption network (RSGA-Net) is constructed for early PD recognition based on OCT images. The extensive experiments on a clinical OCT-PD dataset demonstrate the superiority of RSGA-Net over state-of-the-art methods. Additionally, we provide a visual analysis to explain how retinal structure information affects the decision-making process of DNNs.

帕金森病（PD）是全球主要的神经退行性疾病。精确客观的帕金森病诊断对早期干预和治疗具有重要意义。最近的研究表明，基于光学相干断层扫描（OCT）图像的视网膜结构信息与帕金森病之间存在明显的相关性，这为早期帕金森病识别提供了另一种潜在的方法。然而，如何利用不同视网膜层的视网膜结构信息（如厚度和平均强度）来提高白内障的识别性能，以前还没有人研究过。受上述观察结果的启发，我们首先提出了结构先验知识抽取（SPKE）模块，以获得视网膜结构特征图；然后，我们开发了结构引导和适应注意（SGDA）模块，以根据抽取的视网膜结构特征图充分利用不同视网膜层的潜力。通过在深度神经网络（DNN）的低级阶段嵌入 SPKE 和 SGDA 模块，我们构建了一个视网膜结构引导和适应网络（RSGA-Net），用于基于 OCT 图像的早期 PD 识别。在临床 OCT-PD 数据集上进行的大量实验证明，RSGA-Net 优于最先进的方法。此外，我们还通过视觉分析解释了视网膜结构信息如何影响 DNN 的决策过程。

{"title":"Retinal structure guidance-and-adaption network for early Parkinson’s disease recognition based on OCT images","authors":"Hanfeng Shi , Jiaqi Wei , Richu Jin , Jiaxin Peng , Xingyue Wang , Yan Hu , Xiaoqing Zhang , Jiang Liu","doi":"10.1016/j.compmedimag.2024.102463","DOIUrl":"10.1016/j.compmedimag.2024.102463","url":null,"abstract":"<div><div>Parkinson’s disease (PD) is a leading neurodegenerative disease globally. Precise and objective PD diagnosis is significant for early intervention and treatment. Recent studies have shown significant correlations between retinal structure information and PD based on optical coherence tomography (OCT) images, providing another potential means for early PD recognition. However, how to exploit the retinal structure information (e.g., thickness and mean intensity) from different retinal layers to improve PD recognition performance has not been studied before. Motivated by the above observations, we first propose a structural prior knowledge extraction (SPKE) module to obtain the retinal structure feature maps; then, we develop a structure-guided-and-adaption attention (SGDA) module to fully leverage the potential of different retinal layers based on the extracted retinal structure feature maps. By embedding SPKE and SGDA modules at the low stage of deep neural networks (DNNs), a retinal structure-guided-and-adaption network (RSGA-Net) is constructed for early PD recognition based on OCT images. The extensive experiments on a clinical OCT-PD dataset demonstrate the superiority of RSGA-Net over state-of-the-art methods. Additionally, we provide a visual analysis to explain how retinal structure information affects the decision-making process of DNNs.</div></div>","PeriodicalId":50631,"journal":{"name":"Computerized Medical Imaging and Graphics","volume":"118 ","pages":"Article 102463"},"PeriodicalIF":5.4,"publicationDate":"2024-11-19","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142722393","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Exploratory analysis of Type B Aortic Dissection (TBAD) segmentation in 2D CTA images using various kernels 使用各种核对二维 CTA 图像中 B 型主动脉夹层（TBAD）分割的探索性分析。

IF 5.4 2区医学 Q1 ENGINEERING, BIOMEDICAL

Computerized Medical Imaging and Graphics

Pub Date : 2024-11-18 DOI: 10.1016/j.compmedimag.2024.102460

Ayman Abaid , Srinivas Ilancheran , Talha Iqbal , Niamh Hynes , Ihsan Ullah

Type-B Aortic Dissection is a rare but fatal cardiovascular disease characterized by a tear in the inner layer of the aorta, affecting 3.5 per 100,000 individuals annually. In this work, we explore the feasibility of leveraging two-dimensional Convolutional Neural Network (CNN) models to perform accurate slice-by-slice segmentation of true lumen, false lumen and false lumen thrombus in Computed Tomography Angiography images. The study performed an exploratory analysis of three 2D U-Net models: the baseline 2D U-Net, a variant of U-Net with atrous convolutions, and a U-Net with a custom layer featuring a position-oriented, partially shared weighting scheme kernel. These models were trained and benchmarked against a state-of-the-art baseline 3D U-Net model. Overall, our U-Net with the VGG19 encoder architecture achieved the best performance score among all other models, with a mean Dice score of 80.48% and an IoU score of 72.93%. The segmentation results were also compared with the Segment Anything Model (SAM) and the UniverSeg models. Our findings indicate that our 2D U-Net models excel in false lumen and true lumen segmentation accuracy while achieving lower false lumen thrombus segmentation accuracy compared to the state-of-the-art 3D U-Net model. The study findings highlight the complexities involved in developing segmentation models, especially for cardiovascular medical images, and emphasize the importance of developing lightweight models for real-time decision-making to improve overall patient care.

B 型主动脉夹层是一种以主动脉内层撕裂为特征的罕见但致命的心血管疾病，每年每 10 万人中就有 3.5 人患病。在这项研究中，我们探索了利用二维卷积神经网络（CNN）模型对计算机断层扫描血管造影图像中的真腔、假腔和假腔血栓进行逐片精确分割的可行性。该研究对三种二维 U-Net 模型进行了探索性分析：基线二维 U-Net、带有无齿卷积的 U-Net 变体以及带有自定义层的 U-Net，自定义层的特点是位置导向、部分共享加权方案内核。这些模型都经过了训练，并与最先进的基准 3D U-Net 模型进行了比较。总体而言，我们采用 VGG19 编码器架构的 U-Net 在所有其他模型中取得了最佳性能得分，平均 Dice 得分为 80.48%，IoU 得分为 72.93%。分割结果还与任意分割模型（SAM）和 UniverSeg 模型进行了比较。研究结果表明，与最先进的三维 U-Net 模型相比，我们的二维 U-Net 模型在假腔和真腔分割准确率方面表现出色，而假腔血栓分割准确率较低。研究结果凸显了开发分割模型（尤其是心血管医学图像）的复杂性，并强调了开发轻量级模型用于实时决策以改善整体患者护理的重要性。

{"title":"Exploratory analysis of Type B Aortic Dissection (TBAD) segmentation in 2D CTA images using various kernels","authors":"Ayman Abaid , Srinivas Ilancheran , Talha Iqbal , Niamh Hynes , Ihsan Ullah","doi":"10.1016/j.compmedimag.2024.102460","DOIUrl":"10.1016/j.compmedimag.2024.102460","url":null,"abstract":"<div><div>Type-B Aortic Dissection is a rare but fatal cardiovascular disease characterized by a tear in the inner layer of the aorta, affecting 3.5 per 100,000 individuals annually. In this work, we explore the feasibility of leveraging two-dimensional Convolutional Neural Network (CNN) models to perform accurate slice-by-slice segmentation of true lumen, false lumen and false lumen thrombus in Computed Tomography Angiography images. The study performed an exploratory analysis of three 2D U-Net models: the baseline 2D U-Net, a variant of U-Net with atrous convolutions, and a U-Net with a custom layer featuring a position-oriented, partially shared weighting scheme kernel. These models were trained and benchmarked against a state-of-the-art baseline 3D U-Net model. Overall, our U-Net with the VGG19 encoder architecture achieved the best performance score among all other models, with a mean Dice score of 80.48% and an IoU score of 72.93%. The segmentation results were also compared with the Segment Anything Model (SAM) and the UniverSeg models. Our findings indicate that our 2D U-Net models excel in false lumen and true lumen segmentation accuracy while achieving lower false lumen thrombus segmentation accuracy compared to the state-of-the-art 3D U-Net model. The study findings highlight the complexities involved in developing segmentation models, especially for cardiovascular medical images, and emphasize the importance of developing lightweight models for real-time decision-making to improve overall patient care.</div></div>","PeriodicalId":50631,"journal":{"name":"Computerized Medical Imaging and Graphics","volume":"118 ","pages":"Article 102460"},"PeriodicalIF":5.4,"publicationDate":"2024-11-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142693784","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Exploring transformer reliability in clinically significant prostate cancer segmentation: A comprehensive in-depth investigation 探索具有临床意义的前列腺癌分段中变压器的可靠性：全面深入的调查

IF 5.4 2区医学 Q1 ENGINEERING, BIOMEDICAL

Computerized Medical Imaging and Graphics

Pub Date : 2024-11-17 DOI: 10.1016/j.compmedimag.2024.102459

Gustavo Andrade-Miranda , Pedro Soto Vega , Kamilia Taguelmimt , Hong-Phuong Dang , Dimitris Visvikis , Julien Bert

Despite the growing prominence of transformers in medical image segmentation, their application to clinically significant prostate cancer (csPCa) has been overlooked. Minimal attention has been paid to domain shift analysis and uncertainty assessment, critical for safely implementing computer-aided diagnosis (CAD) systems. Domain shift in medical imagery refers to differences between the data used to train a model and the data evaluated later, arising from variations in imaging equipment, protocols, patient populations, and acquisition noise. While recent models enhance in-domain performance, areas such as robustness and uncertainty estimation in out-of-domain distributions have received limited investigation, creating indecisiveness about model reliability. In contrast, our study addresses csPCa at voxel, lesion, and image levels, investigating models from traditional U-Net to cutting-edge transformers. We focus on four key points: robustness, calibration, out-of-distribution (OOD), and misclassification detection (MD). Findings show that transformer-based models exhibit enhanced robustness at image and lesion levels, both in and out of domain. However, this improvement is not fully translated to the voxel level, where Convolutional Neural Networks (CNNs) outperform in most robustness metrics. Regarding uncertainty, hybrid transformers and transformer encoders performed better, but this trend depends on misclassification or out-of-distribution tasks.

尽管变换器在医学图像分割中的作用日益突出，但其在具有临床意义的前列腺癌（csPCa）中的应用却一直被忽视。人们对域偏移分析和不确定性评估的关注极少，而这对安全实施计算机辅助诊断（CAD）系统至关重要。医学影像中的域偏移指的是用于训练模型的数据与随后评估的数据之间的差异，这种差异是由成像设备、协议、患者群体和采集噪声的变化引起的。虽然最近的模型提高了域内性能，但对域外分布的鲁棒性和不确定性估计等领域的研究却很有限，导致对模型的可靠性举棋不定。相比之下，我们的研究涉及体素、病灶和图像层面的 csPCa，研究了从传统 U-Net 到尖端变换器的各种模型。我们重点关注四个关键点：稳健性、校准、分布外 (OOD) 和误分类检测 (MD)。研究结果表明，基于变换器的模型在图像和病变水平上表现出更强的鲁棒性，无论是在域内还是域外。然而，这种改进并没有完全转化到体素层面，在体素层面，卷积神经网络（CNN）在大多数鲁棒性指标上都表现出色。在不确定性方面，混合变换器和变换器编码器表现更好，但这一趋势取决于误分类或分布外任务。

{"title":"Exploring transformer reliability in clinically significant prostate cancer segmentation: A comprehensive in-depth investigation","authors":"Gustavo Andrade-Miranda , Pedro Soto Vega , Kamilia Taguelmimt , Hong-Phuong Dang , Dimitris Visvikis , Julien Bert","doi":"10.1016/j.compmedimag.2024.102459","DOIUrl":"10.1016/j.compmedimag.2024.102459","url":null,"abstract":"<div><div>Despite the growing prominence of transformers in medical image segmentation, their application to clinically significant prostate cancer (csPCa) has been overlooked. Minimal attention has been paid to domain shift analysis and uncertainty assessment, critical for safely implementing computer-aided diagnosis (CAD) systems. Domain shift in medical imagery refers to differences between the data used to train a model and the data evaluated later, arising from variations in imaging equipment, protocols, patient populations, and acquisition noise. While recent models enhance in-domain performance, areas such as robustness and uncertainty estimation in out-of-domain distributions have received limited investigation, creating indecisiveness about model reliability. In contrast, our study addresses csPCa at voxel, lesion, and image levels, investigating models from traditional U-Net to cutting-edge transformers. We focus on four key points: robustness, calibration, out-of-distribution (OOD), and misclassification detection (MD). Findings show that transformer-based models exhibit enhanced robustness at image and lesion levels, both in and out of domain. However, this improvement is not fully translated to the voxel level, where Convolutional Neural Networks (CNNs) outperform in most robustness metrics. Regarding uncertainty, hybrid transformers and transformer encoders performed better, but this trend depends on misclassification or out-of-distribution tasks.</div></div>","PeriodicalId":50631,"journal":{"name":"Computerized Medical Imaging and Graphics","volume":"118 ","pages":"Article 102459"},"PeriodicalIF":5.4,"publicationDate":"2024-11-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142683103","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

NACNet: A histology context-aware transformer graph convolution network for predicting treatment response to neoadjuvant chemotherapy in Triple Negative Breast Cancer NACNet：用于预测三阴性乳腺癌新辅助化疗治疗反应的组织学上下文感知变换图卷积网络

IF 5.4 2区医学 Q1 ENGINEERING, BIOMEDICAL

Computerized Medical Imaging and Graphics

Pub Date : 2024-11-17 DOI: 10.1016/j.compmedimag.2024.102467

Qiang Li , George Teodoro , Yi Jiang , Jun Kong

Neoadjuvant chemotherapy (NAC) response prediction for triple negative breast cancer (TNBC) patients is a challenging task clinically as it requires understanding complex histology interactions within the tumor microenvironment (TME). Digital whole slide images (WSIs) capture detailed tissue information, but their giga-pixel size necessitates computational methods based on multiple instance learning, which typically analyze small, isolated image tiles without the spatial context of the TME. To address this limitation and incorporate TME spatial histology interactions in predicting NAC response for TNBC patients, we developed a histology context-aware transformer graph convolution network (NACNet). Our deep learning method identifies the histopathological labels on individual image tiles from WSIs, constructs a spatial TME graph, and represents each node with features derived from tissue texture and social network analysis. It predicts NAC response using a transformer graph convolution network model enhanced with graph isomorphism network layers. We evaluate our method with WSIs of a cohort of TNBC patient (N=105) and compared its performance with multiple state-of-the-art machine learning and deep learning models, including both graph and non-graph approaches. Our NACNet achieves 90.0% accuracy, 96.0% sensitivity, 88.0% specificity, and an AUC of 0.82, through eight-fold cross-validation, outperforming baseline models. These comprehensive experimental results suggest that NACNet holds strong potential for stratifying TNBC patients by NAC response, thereby helping to prevent overtreatment, improve patient quality of life, reduce treatment cost, and enhance clinical outcomes, marking an important advancement toward personalized breast cancer treatment.

三阴性乳腺癌（TNBC）患者的新辅助化疗（NAC）反应预测是一项具有挑战性的临床任务，因为它需要了解肿瘤微环境（TME）中复杂的组织学相互作用。数字全切片图像（WSI）能捕捉到详细的组织信息，但其千兆像素的尺寸使得基于多实例学习的计算方法成为必要，这种方法通常分析的是孤立的小块图像，而不考虑肿瘤微环境的空间背景。为了解决这一局限性，并结合TME空间组织学相互作用来预测TNBC患者的NAC反应，我们开发了一种组织学上下文感知变换图卷积网络（NACNet）。我们的深度学习方法从 WSIs 中识别单个图像瓦片上的组织病理学标签，构建空间 TME 图，并用组织纹理和社交网络分析得出的特征来表示每个节点。该方法使用变压器图卷积网络模型预测 NAC 反应，该模型使用图同构网络层进行增强。我们用一组 TNBC 患者（N=105）的 WSI 评估了我们的方法，并将其性能与多种最先进的机器学习和深度学习模型（包括图和非图方法）进行了比较。通过八倍交叉验证，我们的 NACNet 实现了 90.0% 的准确率、96.0% 的灵敏度、88.0% 的特异性和 0.82 的 AUC，表现优于基线模型。这些全面的实验结果表明，NACNet 在根据 NAC 反应对 TNBC 患者进行分层方面具有强大的潜力，从而有助于防止过度治疗、改善患者生活质量、降低治疗成本和提高临床疗效，标志着乳腺癌个性化治疗取得了重要进展。

{"title":"NACNet: A histology context-aware transformer graph convolution network for predicting treatment response to neoadjuvant chemotherapy in Triple Negative Breast Cancer","authors":"Qiang Li , George Teodoro , Yi Jiang , Jun Kong","doi":"10.1016/j.compmedimag.2024.102467","DOIUrl":"10.1016/j.compmedimag.2024.102467","url":null,"abstract":"<div><div>Neoadjuvant chemotherapy (NAC) response prediction for triple negative breast cancer (TNBC) patients is a challenging task clinically as it requires understanding complex histology interactions within the tumor microenvironment (TME). Digital whole slide images (WSIs) capture detailed tissue information, but their giga-pixel size necessitates computational methods based on multiple instance learning, which typically analyze small, isolated image tiles without the spatial context of the TME. To address this limitation and incorporate TME spatial histology interactions in predicting NAC response for TNBC patients, we developed a histology context-aware transformer graph convolution network (NACNet). Our deep learning method identifies the histopathological labels on individual image tiles from WSIs, constructs a spatial TME graph, and represents each node with features derived from tissue texture and social network analysis. It predicts NAC response using a transformer graph convolution network model enhanced with graph isomorphism network layers. We evaluate our method with WSIs of a cohort of TNBC patient (N=105) and compared its performance with multiple state-of-the-art machine learning and deep learning models, including both graph and non-graph approaches. Our NACNet achieves 90.0% accuracy, 96.0% sensitivity, 88.0% specificity, and an AUC of 0.82, through eight-fold cross-validation, outperforming baseline models. These comprehensive experimental results suggest that NACNet holds strong potential for stratifying TNBC patients by NAC response, thereby helping to prevent overtreatment, improve patient quality of life, reduce treatment cost, and enhance clinical outcomes, marking an important advancement toward personalized breast cancer treatment.</div></div>","PeriodicalId":50631,"journal":{"name":"Computerized Medical Imaging and Graphics","volume":"118 ","pages":"Article 102467"},"PeriodicalIF":5.4,"publicationDate":"2024-11-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142700926","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Self-supervised multi-modal feature fusion for predicting early recurrence of hepatocellular carcinoma 预测肝细胞癌早期复发的自我监督多模态特征融合。

IF 5.4 2区医学 Q1 ENGINEERING, BIOMEDICAL

Computerized Medical Imaging and Graphics

Pub Date : 2024-11-14 DOI: 10.1016/j.compmedimag.2024.102457

Sen Wang , Ying Zhao , Jiayi Li , Zongmin Yi , Jun Li , Can Zuo , Yu Yao , Ailian Liu

Surgical resection stands as the primary treatment option for early-stage hepatocellular carcinoma (HCC) patients. Postoperative early recurrence (ER) is a significant factor contributing to the mortality of HCC patients. Therefore, accurately predicting the risk of ER after curative resection is crucial for clinical decision-making and improving patient prognosis. This study leverages a self-supervised multi-modal feature fusion approach, combining multi-phase MRI and clinical features, to predict ER of HCC. Specifically, we utilized attention mechanisms to suppress redundant features, enabling efficient extraction and fusion of multi-phase features. Through self-supervised learning (SSL), we pretrained an encoder on our dataset to extract more generalizable feature representations. Finally, we achieved effective multi-modal information fusion via attention modules. To enhance explainability, we employed Score-CAM to visualize the key regions influencing the model’s predictions. We evaluated the effectiveness of the proposed method on our dataset and found that predictions based on multi-phase feature fusion outperformed those based on single-phase features. Additionally, predictions based on multi-modal feature fusion were superior to those based on single-modal features.

手术切除是早期肝细胞癌（HCC）患者的主要治疗方案。术后早期复发（ER）是导致 HCC 患者死亡的一个重要因素。因此，准确预测治愈性切除术后的早期复发风险对于临床决策和改善患者预后至关重要。本研究利用自监督多模态特征融合方法，结合多相磁共振成像和临床特征，预测 HCC 的 ER。具体来说，我们利用注意力机制抑制冗余特征，从而实现多相特征的高效提取和融合。通过自我监督学习（SSL），我们在数据集上预训练了编码器，以提取更具通用性的特征表征。最后，我们通过注意力模块实现了有效的多模态信息融合。为了提高可解释性，我们采用了 Score-CAM 来可视化影响模型预测的关键区域。我们在数据集上评估了所提方法的有效性，发现基于多相特征融合的预测结果优于基于单相特征的预测结果。此外，基于多模态特征融合的预测结果优于基于单模态特征的预测结果。

{"title":"Self-supervised multi-modal feature fusion for predicting early recurrence of hepatocellular carcinoma","authors":"Sen Wang , Ying Zhao , Jiayi Li , Zongmin Yi , Jun Li , Can Zuo , Yu Yao , Ailian Liu","doi":"10.1016/j.compmedimag.2024.102457","DOIUrl":"10.1016/j.compmedimag.2024.102457","url":null,"abstract":"<div><div>Surgical resection stands as the primary treatment option for early-stage hepatocellular carcinoma (HCC) patients. Postoperative early recurrence (ER) is a significant factor contributing to the mortality of HCC patients. Therefore, accurately predicting the risk of ER after curative resection is crucial for clinical decision-making and improving patient prognosis. This study leverages a self-supervised multi-modal feature fusion approach, combining multi-phase MRI and clinical features, to predict ER of HCC. Specifically, we utilized attention mechanisms to suppress redundant features, enabling efficient extraction and fusion of multi-phase features. Through self-supervised learning (SSL), we pretrained an encoder on our dataset to extract more generalizable feature representations. Finally, we achieved effective multi-modal information fusion via attention modules. To enhance explainability, we employed Score-CAM to visualize the key regions influencing the model’s predictions. We evaluated the effectiveness of the proposed method on our dataset and found that predictions based on multi-phase feature fusion outperformed those based on single-phase features. Additionally, predictions based on multi-modal feature fusion were superior to those based on single-modal features.</div></div>","PeriodicalId":50631,"journal":{"name":"Computerized Medical Imaging and Graphics","volume":"118 ","pages":"Article 102457"},"PeriodicalIF":5.4,"publicationDate":"2024-11-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142689549","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0