Deep learning informed multimodal fusion of radiology and pathology to predict outcomes in HPV-associated oropharyngeal squamous cell carcinoma.

IF 10.8 1区医学 Q1 MEDICINE, RESEARCH & EXPERIMENTAL EBioMedicine Pub Date : 2025-04-01 Epub Date: 2025-03-22 DOI:10.1016/j.ebiom.2025.105663

Bolin Song, Amaury Leroy, Kailin Yang, Tanmoy Dam, Xiangxue Wang, Himanshu Maurya, Tilak Pathak, Jonathan Lee, Sarah Stock, Xiao T Li, Pingfu Fu, Cheng Lu, Paula Toro, Deborah J Chute, Shlomo Koyfman, Nabil F Saba, Mihir R Patel, Anant Madabhushi

{"title":"Deep learning informed multimodal fusion of radiology and pathology to predict outcomes in HPV-associated oropharyngeal squamous cell carcinoma.","authors":"Bolin Song, Amaury Leroy, Kailin Yang, Tanmoy Dam, Xiangxue Wang, Himanshu Maurya, Tilak Pathak, Jonathan Lee, Sarah Stock, Xiao T Li, Pingfu Fu, Cheng Lu, Paula Toro, Deborah J Chute, Shlomo Koyfman, Nabil F Saba, Mihir R Patel, Anant Madabhushi","doi":"10.1016/j.ebiom.2025.105663","DOIUrl":null,"url":null,"abstract":"Background: We aim to predict outcomes of human papillomavirus (HPV)-associated oropharyngeal squamous cell carcinoma (OPSCC), a subtype of head and neck cancer characterized with improved clinical outcome and better response to therapy. Pathology and radiology focused AI-based prognostic models have been independently developed for OPSCC, but their integration incorporating both primary tumour (PT) and metastatic cervical lymph node (LN) remains unexamined.Methods: We investigate the prognostic value of an AI approach termed the swintransformer-based multimodal and multi-region data fusion framework (SMuRF). SMuRF integrates features from CT corresponding to the PT and LN, as well as whole slide pathology images from the PT as a predictor of survival and tumour grade in HPV-associated OPSCC. SMuRF employs cross-modality and cross-region window based multi-head self-attention mechanisms to capture interactions between features across tumour habitats and image scales.Findings: Developed and tested on a cohort of 277 patients with OPSCC with matched radiology and pathology images, SMuRF demonstrated strong performance (C-index = 0.81 for DFS prediction and AUC = 0.75 for tumour grade classification) and emerged as an independent prognostic biomarker for DFS (hazard ratio [HR] = 17, 95% confidence interval [CI], 4.9-58, p < 0.0001) and tumour grade (odds ratio [OR] = 3.7, 95% CI, 1.4-10.5, p = 0.01) controlling for other clinical variables (i.e., T-, N-stage, age, smoking, sex and treatment modalities). Importantly, SMuRF outperformed unimodal models derived from radiology or pathology alone.Interpretation: Our findings underscore the potential of multimodal deep learning in accurately stratifying OPSCC risk, informing tailored treatment strategies and potentially refining existing treatment algorithms.Funding: The National Institutes of Health, the U.S. Department of Veterans Affairs and National Institute of Biomedical Imaging and Bioengineering.","PeriodicalId":11494,"journal":{"name":"EBioMedicine","volume":"114 ","pages":"105663"},"PeriodicalIF":10.8000,"publicationDate":"2025-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11979917/pdf/","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"EBioMedicine","FirstCategoryId":"3","ListUrlMain":"https://doi.org/10.1016/j.ebiom.2025.105663","RegionNum":1,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"2025/3/22 0:00:00","PubModel":"Epub","JCR":"Q1","JCRName":"MEDICINE, RESEARCH & EXPERIMENTAL","Score":null,"Total":0}

引用次数: 0

Abstract

Background: We aim to predict outcomes of human papillomavirus (HPV)-associated oropharyngeal squamous cell carcinoma (OPSCC), a subtype of head and neck cancer characterized with improved clinical outcome and better response to therapy. Pathology and radiology focused AI-based prognostic models have been independently developed for OPSCC, but their integration incorporating both primary tumour (PT) and metastatic cervical lymph node (LN) remains unexamined.

Methods: We investigate the prognostic value of an AI approach termed the swintransformer-based multimodal and multi-region data fusion framework (SMuRF). SMuRF integrates features from CT corresponding to the PT and LN, as well as whole slide pathology images from the PT as a predictor of survival and tumour grade in HPV-associated OPSCC. SMuRF employs cross-modality and cross-region window based multi-head self-attention mechanisms to capture interactions between features across tumour habitats and image scales.

Findings: Developed and tested on a cohort of 277 patients with OPSCC with matched radiology and pathology images, SMuRF demonstrated strong performance (C-index = 0.81 for DFS prediction and AUC = 0.75 for tumour grade classification) and emerged as an independent prognostic biomarker for DFS (hazard ratio [HR] = 17, 95% confidence interval [CI], 4.9-58, p < 0.0001) and tumour grade (odds ratio [OR] = 3.7, 95% CI, 1.4-10.5, p = 0.01) controlling for other clinical variables (i.e., T-, N-stage, age, smoking, sex and treatment modalities). Importantly, SMuRF outperformed unimodal models derived from radiology or pathology alone.

Interpretation: Our findings underscore the potential of multimodal deep learning in accurately stratifying OPSCC risk, informing tailored treatment strategies and potentially refining existing treatment algorithms.

Funding: The National Institutes of Health, the U.S. Department of Veterans Affairs and National Institute of Biomedical Imaging and Bioengineering.

查看原文

微信好友朋友圈 QQ好友复制链接

本刊更多论文

通过深度学习对放射学和病理学进行多模态融合，预测与人乳头瘤病毒相关的口咽鳞状细胞癌的预后。

背景：我们的目的是预测人乳头瘤病毒（HPV）相关口咽鳞状细胞癌（OPSCC）的预后，这是头颈癌的一种亚型，其特点是临床预后改善，对治疗的反应更好。基于人工智能的OPSCC病理学和放射学预后模型已经独立开发，但其整合原发肿瘤（PT）和转移性颈淋巴结（LN）仍未得到检验。方法：我们研究了一种称为基于涡流变压器的多模式和多区域数据融合框架（SMuRF）的人工智能方法的预测价值。SMuRF整合了来自PT和LN对应的CT特征，以及来自PT的整个病理切片图像，作为hpv相关OPSCC的生存和肿瘤分级的预测因子。SMuRF采用基于跨模态和跨区域窗口的多头自注意机制来捕获跨肿瘤栖息地和图像尺度的特征之间的相互作用。研究结果：SMuRF在277例具有匹配放射学和病理学图像的OPSCC患者队列中开发和测试，显示出强大的性能（预测DFS的c指数= 0.81，肿瘤分级的AUC = 0.75），并成为DFS的独立预后生物标志物（风险比[HR] = 17, 95%置信区间[CI]， 4.9-58, p）。我们的研究结果强调了多模态深度学习在准确分层OPSCC风险、提供量身定制的治疗策略和潜在地改进现有治疗算法方面的潜力。资助：国家卫生研究院，美国退伍军人事务部和国家生物医学成像和生物工程研究所。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文去求助

来源期刊

EBioMedicine Biochemistry, Genetics and Molecular Biology-General Biochemistry,Genetics and Molecular Biology

CiteScore

17.70

自引率

0.90%

发文量

579

审稿时长

5 weeks

期刊介绍： eBioMedicine is a comprehensive biomedical research journal that covers a wide range of studies that are relevant to human health. Our focus is on original research that explores the fundamental factors influencing human health and disease, including the discovery of new therapeutic targets and treatments, the identification of biomarkers and diagnostic tools, and the investigation and modification of disease pathways and mechanisms. We welcome studies from any biomedical discipline that contribute to our understanding of disease and aim to improve human health.