Pub Date : 2024-08-01DOI: 10.1016/S2589-7500(24)00095-5
The development and commercialisation of medical decision systems based on artificial intelligence (AI) far outpaces our understanding of their value for clinicians. Although applicable across many forms of medicine, we focus on characterising the diagnostic decisions of radiologists through the concept of ecologically bounded reasoning, review the differences between clinician decision making and medical AI model decision making, and reveal how these differences pose fundamental challenges for integrating AI into radiology. We argue that clinicians are contextually motivated, mentally resourceful decision makers, whereas AI models are contextually stripped, correlational decision makers, and discuss misconceptions about clinician–AI interaction stemming from this misalignment of capabilities. We outline how future research on clinician–AI interaction could better address the cognitive considerations of decision making and be used to enhance the safety and usability of AI models in high-risk medical decision-making contexts.
{"title":"Medical artificial intelligence for clinicians: the lost cognitive perspective","authors":"","doi":"10.1016/S2589-7500(24)00095-5","DOIUrl":"10.1016/S2589-7500(24)00095-5","url":null,"abstract":"<div><p>The development and commercialisation of medical decision systems based on artificial intelligence (AI) far outpaces our understanding of their value for clinicians. Although applicable across many forms of medicine, we focus on characterising the diagnostic decisions of radiologists through the concept of ecologically bounded reasoning, review the differences between clinician decision making and medical AI model decision making, and reveal how these differences pose fundamental challenges for integrating AI into radiology. We argue that clinicians are contextually motivated, mentally resourceful decision makers, whereas AI models are contextually stripped, correlational decision makers, and discuss misconceptions about clinician–AI interaction stemming from this misalignment of capabilities. We outline how future research on clinician–AI interaction could better address the cognitive considerations of decision making and be used to enhance the safety and usability of AI models in high-risk medical decision-making contexts.</p></div>","PeriodicalId":48534,"journal":{"name":"Lancet Digital Health","volume":null,"pages":null},"PeriodicalIF":23.8,"publicationDate":"2024-08-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.sciencedirect.com/science/article/pii/S2589750024000955/pdfft?md5=c4262279ee0696247e86b8dc47f4a153&pid=1-s2.0-S2589750024000955-main.pdf","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141767680","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-08-01DOI: 10.1016/S2589-7500(24)00116-X
Background
The density of tumour-infiltrating lymphocytes (TILs) could be prognostic in ductal carcinoma in situ (DCIS). However, manual TIL quantification is time-consuming and suffers from interobserver and intraobserver variability. In this study, we developed a TIL-based computational pathology biomarker and evaluated its association with the risk of recurrence and benefit of adjuvant treatment in a clinical trial cohort.
Methods
In this retrospective cohort study, a computational pathology pipeline was developed to generate a TIL-based biomarker (CPath TIL categories). Subsequently, the signature underwent a masked independent validation on H&E-stained whole-section images of 755 patients with DCIS from the UK/ANZ DCIS randomised controlled trial. Specifically, continuous biomarker CPath TIL score was calculated as the average TIL density in the DCIS microenvironment and dichotomised into binary biomarker CPath TIL categories (CPath TIL-high vs CPath TIL-low) using the median value as a cutoff. The primary outcome was ipsilateral breast event (IBE; either recurrence of DCIS [DCIS-IBE] or invasive progression [I-IBE]). The Cox proportional hazards model was used to estimate the hazard ratio (HR).
Findings
CPath TIL-score was evaluable in 718 (95%) of 755 patients (151 IBEs). Patients with CPath TIL-high DCIS had a greater risk of IBE than those with CPath TIL-low DCIS (HR 2·10 [95% CI 1·39–3·18]; p=0·0004). The risk of I-IBE was greater in patients with CPath TIL-high DCIS than those with CPath TIL-low DCIS (3·09 [1·56–6·14]; p=0·0013), and the risk of DCIS-IBE was non-significantly higher in those with CPath TIL-high DCIS (1·61 [0·95–2·72]; p=0·077). A significant interaction (pinteraction=0·025) between CPath TIL categories and radiotherapy was observed with a greater magnitude of radiotherapy benefit in preventing IBE in CPath TIL-high DCIS (0·32 [0·19–0·54]) than CPath TIL-low DCIS (0·40 [0·20–0·81]).
Interpretation
High TIL density is associated with higher recurrence risk—particularly of invasive recurrence—and greater radiotherapy benefit in patients with DCIS. Our TIL-based computational pathology signature has a prognostic and predictive role in DCIS.
Funding
National Cancer Institute under award number U01CA269181, Cancer Research UK (C569/A12061; C569/A16891), and the Breast Cancer Research Foundation, New York (NY, USA).
背景：肿瘤浸润淋巴细胞（TIL）的密度可作为导管原位癌（DCIS）的预后指标。然而，人工定量 TIL 不仅耗时，而且存在观察者之间和观察者内部的差异。在本研究中，我们开发了一种基于TIL的计算病理学生物标志物，并在临床试验队列中评估了其与复发风险和辅助治疗获益的相关性：在这项回顾性队列研究中，开发了一个计算病理学管道，以生成基于TIL的生物标志物（CPath TIL类别）。随后，对英国/新西兰 DCIS 随机对照试验中 755 名 DCIS 患者的 H&E 染色全切片图像进行了独立的掩蔽验证。具体来说，连续生物标志物 CPath TIL 评分计算为 DCIS 微环境中的平均 TIL 密度，并以中值作为分界点，将其分为二元生物标志物 CPath TIL 类别（CPath TIL 高 vs CPath TIL 低）。主要结果是同侧乳腺事件（IBE；DCIS复发[DCIS-IBE]或浸润性进展[I-IBE]）。采用 Cox 比例危险模型估算危险比 (HR)：在 755 例患者（151 例 IBE）中，有 718 例（95%）的 CPath TIL 评分可进行评估。CPath TIL 高的 DCIS 患者比 CPath TIL 低的 DCIS 患者发生 IBE 的风险更高（HR 2-10 [95% CI 1-39-3-18]; p=0-0004）。CPath TIL高的DCIS患者发生I-BE的风险高于CPath TIL低的DCIS患者（3-09 [1-56-6-14]; p=0-0013），CPath TIL高的DCIS患者发生DCIS-IBE的风险无显著性差异（1-61 [0-95-2-72]; p=0-077）。CPath TIL类别与放疗之间存在明显的交互作用（pinteraction=0-025），CPath TIL高的DCIS（0-32 [0-19-0-54]）比CPath TIL低的DCIS（0-40 [0-20-0-81]）在预防IBE方面的放疗获益更大：高TIL密度与DCIS患者较高的复发风险（尤其是侵袭性复发）和更大的放疗获益相关。我们基于TIL的计算病理学特征对DCIS具有预后和预测作用：美国国立癌症研究所（获奖号：U01CA269181）、英国癌症研究中心（C569/A12061; C569/A16891）和美国纽约乳腺癌研究基金会。
{"title":"A prognostic and predictive computational pathology immune signature for ductal carcinoma in situ: retrospective results from a cohort within the UK/ANZ DCIS trial","authors":"","doi":"10.1016/S2589-7500(24)00116-X","DOIUrl":"10.1016/S2589-7500(24)00116-X","url":null,"abstract":"<div><h3>Background</h3><p>The density of tumour-infiltrating lymphocytes (TILs) could be prognostic in ductal carcinoma in situ (DCIS). However, manual TIL quantification is time-consuming and suffers from interobserver and intraobserver variability. In this study, we developed a TIL-based computational pathology biomarker and evaluated its association with the risk of recurrence and benefit of adjuvant treatment in a clinical trial cohort.</p></div><div><h3>Methods</h3><p>In this retrospective cohort study, a computational pathology pipeline was developed to generate a TIL-based biomarker (CPath TIL categories). Subsequently, the signature underwent a masked independent validation on H&E-stained whole-section images of 755 patients with DCIS from the UK/ANZ DCIS randomised controlled trial. Specifically, continuous biomarker CPath TIL score was calculated as the average TIL density in the DCIS microenvironment and dichotomised into binary biomarker CPath TIL categories (CPath TIL-high <em>vs</em> CPath TIL-low) using the median value as a cutoff. The primary outcome was ipsilateral breast event (IBE; either recurrence of DCIS [DCIS-IBE] or invasive progression [I-IBE]). The Cox proportional hazards model was used to estimate the hazard ratio (HR).</p></div><div><h3>Findings</h3><p>CPath TIL-score was evaluable in 718 (95%) of 755 patients (151 IBEs). Patients with CPath TIL-high DCIS had a greater risk of IBE than those with CPath TIL-low DCIS (HR 2·10 [95% CI 1·39–3·18]; p=0·0004). The risk of I-IBE was greater in patients with CPath TIL-high DCIS than those with CPath TIL-low DCIS (3·09 [1·56–6·14]; p=0·0013), and the risk of DCIS-IBE was non-significantly higher in those with CPath TIL-high DCIS (1·61 [0·95–2·72]; p=0·077). A significant interaction (p<sub>interaction</sub>=0·025) between CPath TIL categories and radiotherapy was observed with a greater magnitude of radiotherapy benefit in preventing IBE in CPath TIL-high DCIS (0·32 [0·19–0·54]) than CPath TIL-low DCIS (0·40 [0·20–0·81]).</p></div><div><h3>Interpretation</h3><p>High TIL density is associated with higher recurrence risk—particularly of invasive recurrence—and greater radiotherapy benefit in patients with DCIS. Our TIL-based computational pathology signature has a prognostic and predictive role in DCIS.</p></div><div><h3>Funding</h3><p>National Cancer Institute under award number U01CA269181, Cancer Research UK (C569/A12061; C569/A16891), and the Breast Cancer Research Foundation, New York (NY, USA).</p></div>","PeriodicalId":48534,"journal":{"name":"Lancet Digital Health","volume":null,"pages":null},"PeriodicalIF":23.8,"publicationDate":"2024-08-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.sciencedirect.com/science/article/pii/S258975002400116X/pdfft?md5=995a38719dfb36fc24e8288708c57372&pid=1-s2.0-S258975002400116X-main.pdf","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141581170","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-08-01DOI: 10.1016/S2589-7500(24)00156-0
{"title":"Correction to Lancet Digit Health 2024; 6: e562–69","authors":"","doi":"10.1016/S2589-7500(24)00156-0","DOIUrl":"10.1016/S2589-7500(24)00156-0","url":null,"abstract":"","PeriodicalId":48534,"journal":{"name":"Lancet Digital Health","volume":null,"pages":null},"PeriodicalIF":23.8,"publicationDate":"2024-08-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.sciencedirect.com/science/article/pii/S2589750024001560/pdfft?md5=499968d7a55949c6abf6cf414a476422&pid=1-s2.0-S2589750024001560-main.pdf","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141767676","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-08-01DOI: 10.1016/S2589-7500(24)00144-4
{"title":"Evaluation and communication of pandemic scenarios","authors":"","doi":"10.1016/S2589-7500(24)00144-4","DOIUrl":"10.1016/S2589-7500(24)00144-4","url":null,"abstract":"","PeriodicalId":48534,"journal":{"name":"Lancet Digital Health","volume":null,"pages":null},"PeriodicalIF":23.8,"publicationDate":"2024-08-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.sciencedirect.com/science/article/pii/S2589750024001444/pdfft?md5=208ba0fe86d5c6cdff7a0fec55078935&pid=1-s2.0-S2589750024001444-main.pdf","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141767677","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-08-01DOI: 10.1016/S2589-7500(24)00096-7
Background
Early identification of an acute respiratory infection is important for reducing transmission and enabling earlier therapeutic intervention. We aimed to prospectively evaluate the feasibility of home-based diagnostic self-testing of viral pathogens in individuals prompted to do so on the basis of self-reported symptoms or individual changes in physiological parameters detected via a wearable sensor.
Methods
DETECT-AHEAD was a prospective, decentralised, randomised controlled trial carried out in a subpopulation of an existing cohort (DETECT) of individuals enrolled in a digital-only observational study in the USA. Participants aged 18 years or older were randomly assigned (1:1:1) with a block randomisation scheme stratified by under-represented in biomedical research status. All participants were offered a wearable sensor (Fitbit Sense smartwatch). Participants in groups 1 and 2 received an at-home self-test kit (Alveo be.well) for two acute respiratory viral pathogens: SARS-CoV-2 and respiratory syncytial virus. Participants in group 1 could be alerted through the DETECT study app to take the at-home test on the basis of changes in their physiological data (as detected by our algorithm) or due to self-reported symptoms; those in group 2 were prompted via the app to self-test only due to symptoms. Group 3 served as the control group, without alerts or home testing capability. The primary endpoints, assessed on an intention-to-treat basis, were the number of acute respiratory infections presented (self-reported) and diagnosed (electronic health record), and the number of participants using at-home testing in groups 1 and 2. This trial is registered with ClinicalTrials.gov, NCT04336020.
Findings
Between Sept 28 and Dec 30, 2021, 450 participants were recruited and randomly assigned to group 1 (n=149), group 2 (n=151), or group 3 (n=150). 179 (40%) participants were male, 264 (59%) were female, and seven (2%) identified as other. 232 (52%) were from populations historically under-represented in biomedical research. 118 (39%) of the 300 participants in groups 1 and 2 were prompted to self-test, with 61 (52%) successfully completing self-testing. Participants were prompted to home-test more frequently due to symptoms (41 [28%] in group 1 and 51 [34%] in group 2) than due to detected physiological changes (26 [17%] in group 1). Significantly more participants in group 1 received alerts to test than did those in group 2 (67 [45%] vs 51 [34%]; p=0·047). Of the 61 individuals who were prompted to test and successfully did so, 19 (31%) tested positive for a viral pathogen—all for SARS-CoV-2. The individuals diagnosed as positive for SARS-CoV-2 in the electronic health record were eight (5%) in group 1, four (3%) in group 2, and two (1%) in group 3, but it was difficult to c
{"title":"Feasibility of wearable sensor signals and self-reported symptoms to prompt at-home testing for acute respiratory viruses in the USA (DETECT-AHEAD): a decentralised, randomised controlled trial","authors":"","doi":"10.1016/S2589-7500(24)00096-7","DOIUrl":"10.1016/S2589-7500(24)00096-7","url":null,"abstract":"<div><h3>Background</h3><p>Early identification of an acute respiratory infection is important for reducing transmission and enabling earlier therapeutic intervention. We aimed to prospectively evaluate the feasibility of home-based diagnostic self-testing of viral pathogens in individuals prompted to do so on the basis of self-reported symptoms or individual changes in physiological parameters detected via a wearable sensor.</p></div><div><h3>Methods</h3><p>DETECT-AHEAD was a prospective, decentralised, randomised controlled trial carried out in a subpopulation of an existing cohort (DETECT) of individuals enrolled in a digital-only observational study in the USA. Participants aged 18 years or older were randomly assigned (1:1:1) with a block randomisation scheme stratified by under-represented in biomedical research status. All participants were offered a wearable sensor (Fitbit Sense smartwatch). Participants in groups 1 and 2 received an at-home self-test kit (Alveo be.well) for two acute respiratory viral pathogens: SARS-CoV-2 and respiratory syncytial virus. Participants in group 1 could be alerted through the DETECT study app to take the at-home test on the basis of changes in their physiological data (as detected by our algorithm) or due to self-reported symptoms; those in group 2 were prompted via the app to self-test only due to symptoms. Group 3 served as the control group, without alerts or home testing capability. The primary endpoints, assessed on an intention-to-treat basis, were the number of acute respiratory infections presented (self-reported) and diagnosed (electronic health record), and the number of participants using at-home testing in groups 1 and 2. This trial is registered with <span><span>ClinicalTrials.gov</span><svg><path></path></svg></span>, <span><span>NCT04336020</span><svg><path></path></svg></span>.</p></div><div><h3>Findings</h3><p>Between Sept 28 and Dec 30, 2021, 450 participants were recruited and randomly assigned to group 1 (n=149), group 2 (n=151), or group 3 (n=150). 179 (40%) participants were male, 264 (59%) were female, and seven (2%) identified as other. 232 (52%) were from populations historically under-represented in biomedical research. 118 (39%) of the 300 participants in groups 1 and 2 were prompted to self-test, with 61 (52%) successfully completing self-testing. Participants were prompted to home-test more frequently due to symptoms (41 [28%] in group 1 and 51 [34%] in group 2) than due to detected physiological changes (26 [17%] in group 1). Significantly more participants in group 1 received alerts to test than did those in group 2 (67 [45%] <em>vs</em> 51 [34%]; p=0·047). Of the 61 individuals who were prompted to test and successfully did so, 19 (31%) tested positive for a viral pathogen—all for SARS-CoV-2. The individuals diagnosed as positive for SARS-CoV-2 in the electronic health record were eight (5%) in group 1, four (3%) in group 2, and two (1%) in group 3, but it was difficult to c","PeriodicalId":48534,"journal":{"name":"Lancet Digital Health","volume":null,"pages":null},"PeriodicalIF":23.8,"publicationDate":"2024-08-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11296689/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141767678","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-08-01DOI: 10.1016/S2589-7500(24)00157-2
{"title":"Pathology in the era of generative AI","authors":"","doi":"10.1016/S2589-7500(24)00157-2","DOIUrl":"10.1016/S2589-7500(24)00157-2","url":null,"abstract":"","PeriodicalId":48534,"journal":{"name":"Lancet Digital Health","volume":null,"pages":null},"PeriodicalIF":23.8,"publicationDate":"2024-08-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.sciencedirect.com/science/article/pii/S2589750024001572/pdfft?md5=b34ff67d1c10eff92c7810c03c24a9b8&pid=1-s2.0-S2589750024001572-main.pdf","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141767682","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-08-01DOI: 10.1016/S2589-7500(24)00114-6
The rapid evolution of generative artificial intelligence (AI) models including OpenAI's ChatGPT signals a promising era for medical research. In this Viewpoint, we explore the integration and challenges of large language models (LLMs) in digital pathology, a rapidly evolving domain demanding intricate contextual understanding. The restricted domain-specific efficiency of LLMs necessitates the advent of tailored AI tools, as illustrated by advancements seen in the last few years including FrugalGPT and BioBERT. Our initiative in digital pathology emphasises the potential of domain-specific AI tools, where a curated literature database coupled with a user-interactive web application facilitates precise, referenced information retrieval. Motivated by the success of this initiative, we discuss how domain-specific approaches substantially minimise the risk of inaccurate responses, enhancing the reliability and accuracy of information extraction. We also highlight the broader implications of such tools, particularly in streamlining access to scientific research and democratising access to computational pathology techniques for scientists with little coding experience. This Viewpoint calls for an enhanced integration of domain-specific text-generation AI tools in academic settings to facilitate continuous learning and adaptation to the dynamically evolving landscape of medical research.
{"title":"ChatGPT for digital pathology research","authors":"","doi":"10.1016/S2589-7500(24)00114-6","DOIUrl":"10.1016/S2589-7500(24)00114-6","url":null,"abstract":"<div><p>The rapid evolution of generative artificial intelligence (AI) models including OpenAI's ChatGPT signals a promising era for medical research. In this Viewpoint, we explore the integration and challenges of large language models (LLMs) in digital pathology, a rapidly evolving domain demanding intricate contextual understanding. The restricted domain-specific efficiency of LLMs necessitates the advent of tailored AI tools, as illustrated by advancements seen in the last few years including FrugalGPT and BioBERT. Our initiative in digital pathology emphasises the potential of domain-specific AI tools, where a curated literature database coupled with a user-interactive web application facilitates precise, referenced information retrieval. Motivated by the success of this initiative, we discuss how domain-specific approaches substantially minimise the risk of inaccurate responses, enhancing the reliability and accuracy of information extraction. We also highlight the broader implications of such tools, particularly in streamlining access to scientific research and democratising access to computational pathology techniques for scientists with little coding experience. This Viewpoint calls for an enhanced integration of domain-specific text-generation AI tools in academic settings to facilitate continuous learning and adaptation to the dynamically evolving landscape of medical research.</p></div>","PeriodicalId":48534,"journal":{"name":"Lancet Digital Health","volume":null,"pages":null},"PeriodicalIF":23.8,"publicationDate":"2024-08-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11299190/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141581171","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-08-01DOI: 10.1016/S2589-7500(24)00112-2
{"title":"Migration background, skin colour, gender, and infectious disease presentation in clinical vignettes","authors":"","doi":"10.1016/S2589-7500(24)00112-2","DOIUrl":"10.1016/S2589-7500(24)00112-2","url":null,"abstract":"","PeriodicalId":48534,"journal":{"name":"Lancet Digital Health","volume":null,"pages":null},"PeriodicalIF":23.8,"publicationDate":"2024-08-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.sciencedirect.com/science/article/pii/S2589750024001122/pdfft?md5=79136ea6973726c095e2ef5ef718b584&pid=1-s2.0-S2589750024001122-main.pdf","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141767681","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-08-01DOI: 10.1016/S2589-7500(24)00097-9
Background
Artificial intelligence (AI) applications in health care have been effective in many areas of medicine, but they are often trained for a single task using labelled data, making deployment and generalisability challenging. How well a general-purpose AI language model performs diagnosis and triage relative to physicians and laypeople is not well understood.
Methods
We compared the predictive accuracy of Generative Pre-trained Transformer 3 (GPT-3)'s diagnostic and triage ability for 48 validated synthetic case vignettes (<50 words; sixth-grade reading level or below) of both common (eg, viral illness) and severe (eg, heart attack) conditions to a nationally representative sample of 5000 lay people from the USA who could use the internet to find the correct options and 21 practising physicians at Harvard Medical School. There were 12 vignettes for each of four triage categories: emergent, within one day, within 1 week, and self-care. The correct diagnosis and triage category (ie, ground truth) for each vignette was determined by two general internists at Harvard Medical School. For each vignette, human respondents and GPT-3 were prompted to list diagnoses in order of likelihood, and the vignette was marked as correct if the ground-truth diagnosis was in the top three of the listed diagnoses. For triage accuracy, we examined whether the human respondents’ and GPT-3's selected triage was exactly correct according to the four triage categories, or matched a dichotomised triage variable (emergent or within 1 day vs within 1 week or self-care). We estimated GPT-3's diagnostic and triage confidence on a given vignette using a modified bootstrap resampling procedure, and examined how well calibrated GPT-3's confidence was by computing calibration curves and Brier scores. We also performed subgroup analysis by case acuity, and an error analysis for triage advice to characterise how its advice might affect patients using this tool to decide if they should seek medical care immediately.
Findings
Among all cases, GPT-3 replied with the correct diagnosis in its top three for 88% (42/48, 95% CI 75–94) of cases, compared with 54% (2700/5000, 53–55) for lay individuals (p<0.0001) and 96% (637/666, 94–97) for physicians (p=0·012). GPT-3 triaged 70% correct (34/48, 57–82) versus 74% (3706/5000, 73–75; p=0.60) for lay individuals and 91% (608/666, 89–93%; p<0.0001) for physicians. As measured by the Brier score, GPT-3 confidence in its top prediction was reasonably well calibrated for diagnosis (Brier score=0·18) and triage (Brier score=0·22). We observed an inverse relationship between case acuity and GPT-3 accuracy (p<0·0001) with a fitted trend line of –8·33% decrease in accuracy for every level of increase in case acuity. For triage error analysis, GPT-3 deprioritised truly emergent cases in seven instances.
{"title":"The diagnostic and triage accuracy of the GPT-3 artificial intelligence model: an observational study","authors":"","doi":"10.1016/S2589-7500(24)00097-9","DOIUrl":"10.1016/S2589-7500(24)00097-9","url":null,"abstract":"<div><h3>Background</h3><p>Artificial intelligence (AI) applications in health care have been effective in many areas of medicine, but they are often trained for a single task using labelled data, making deployment and generalisability challenging. How well a general-purpose AI language model performs diagnosis and triage relative to physicians and laypeople is not well understood.</p></div><div><h3>Methods</h3><p>We compared the predictive accuracy of Generative Pre-trained Transformer 3 (GPT-3)'s diagnostic and triage ability for 48 validated synthetic case vignettes (<50 words; sixth-grade reading level or below) of both common (eg, viral illness) and severe (eg, heart attack) conditions to a nationally representative sample of 5000 lay people from the USA who could use the internet to find the correct options and 21 practising physicians at Harvard Medical School. There were 12 vignettes for each of four triage categories: emergent, within one day, within 1 week, and self-care. The correct diagnosis and triage category (ie, ground truth) for each vignette was determined by two general internists at Harvard Medical School. For each vignette, human respondents and GPT-3 were prompted to list diagnoses in order of likelihood, and the vignette was marked as correct if the ground-truth diagnosis was in the top three of the listed diagnoses. For triage accuracy, we examined whether the human respondents’ and GPT-3's selected triage was exactly correct according to the four triage categories, or matched a dichotomised triage variable (emergent or within 1 day <em>vs</em> within 1 week or self-care). We estimated GPT-3's diagnostic and triage confidence on a given vignette using a modified bootstrap resampling procedure, and examined how well calibrated GPT-3's confidence was by computing calibration curves and Brier scores. We also performed subgroup analysis by case acuity, and an error analysis for triage advice to characterise how its advice might affect patients using this tool to decide if they should seek medical care immediately.</p></div><div><h3>Findings</h3><p>Among all cases, GPT-3 replied with the correct diagnosis in its top three for 88% (42/48, 95% CI 75–94) of cases, compared with 54% (2700/5000, 53–55) for lay individuals (p<0.0001) and 96% (637/666, 94–97) for physicians (p=0·012). GPT-3 triaged 70% correct (34/48, 57–82) versus 74% (3706/5000, 73–75; p=0.60) for lay individuals and 91% (608/666, 89–93%; p<0.0001) for physicians. As measured by the Brier score, GPT-3 confidence in its top prediction was reasonably well calibrated for diagnosis (Brier score=0·18) and triage (Brier score=0·22). We observed an inverse relationship between case acuity and GPT-3 accuracy (p<0·0001) with a fitted trend line of –8·33% decrease in accuracy for every level of increase in case acuity. For triage error analysis, GPT-3 deprioritised truly emergent cases in seven instances.</p></div><div><h3>Interpretation</h3><p>A general-purpose A","PeriodicalId":48534,"journal":{"name":"Lancet Digital Health","volume":null,"pages":null},"PeriodicalIF":23.8,"publicationDate":"2024-08-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.sciencedirect.com/science/article/pii/S2589750024000979/pdfft?md5=ea4e50c92b21c03fc0e3ebee146bfe6e&pid=1-s2.0-S2589750024000979-main.pdf","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141767684","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-07-19DOI: 10.1016/S2589-7500(24)00118-3
Background
Computer-aided detection (CAD) can help identify people with active tuberculosis left undetected. However, few studies have compared the performance of commercially available CAD products for screening in high tuberculosis and high HIV settings, and there is poor understanding of threshold selection across products in different populations. We aimed to compare CAD products' performance, with further analyses on subgroup performance and threshold selection.
Methods
We evaluated 12 CAD products on a case–control sample of participants from a South African tuberculosis prevalence survey. Only those with microbiological test results were eligible. The primary outcome was comparing products' accuracy using the area under the receiver operating characteristic curve (AUC) against microbiological evidence. Threshold analyses were performed based on pre-defined criteria and across all thresholds. We conducted subgroup analyses including age, gender, HIV status, previous tuberculosis history, symptoms presence, and current smoking status.
Findings
Of the 774 people included, 516 were bacteriologically negative and 258 were bacteriologically positive. Diverse accuracy was noted: Lunit and Nexus had AUCs near 0·9, followed by qXR, JF CXR-2, InferRead, Xvision, and ChestEye (AUCs 0·8–0·9). XrayAME, RADIFY, and TiSepX-TB had AUC under 0·8. Thresholds varied notably across these products and different versions of the same products. Certain products (Lunit, Nexus, JF CXR-2, and qXR) maintained high sensitivity (>90%) across a wide threshold range while reducing the number of individuals requiring confirmatory diagnostic testing. All products generally performed worst in older individuals, people with previous tuberculosis, and people with HIV. Variations in thresholds, sensitivity, and specificity existed across groups and settings.
Interpretation
Several previously unevaluated products performed similarly to those evaluated by WHO. Thresholds differed across products and demographic subgroups. The rapid emergence of products and versions necessitates a global strategy to validate new versions and software to support CAD product and threshold selections.
{"title":"Computer-aided detection of tuberculosis from chest radiographs in a tuberculosis prevalence survey in South Africa: external validation and modelled impacts of commercially available artificial intelligence software","authors":"","doi":"10.1016/S2589-7500(24)00118-3","DOIUrl":"10.1016/S2589-7500(24)00118-3","url":null,"abstract":"<div><h3>Background</h3><p>Computer-aided detection (CAD) can help identify people with active tuberculosis left undetected. However, few studies have compared the performance of commercially available CAD products for screening in high tuberculosis and high HIV settings, and there is poor understanding of threshold selection across products in different populations. We aimed to compare CAD products' performance, with further analyses on subgroup performance and threshold selection.</p></div><div><h3>Methods</h3><p>We evaluated 12 CAD products on a case–control sample of participants from a South African tuberculosis prevalence survey. Only those with microbiological test results were eligible. The primary outcome was comparing products' accuracy using the area under the receiver operating characteristic curve (AUC) against microbiological evidence. Threshold analyses were performed based on pre-defined criteria and across all thresholds. We conducted subgroup analyses including age, gender, HIV status, previous tuberculosis history, symptoms presence, and current smoking status.</p></div><div><h3>Findings</h3><p>Of the 774 people included, 516 were bacteriologically negative and 258 were bacteriologically positive. Diverse accuracy was noted: Lunit and Nexus had AUCs near 0·9, followed by qXR, JF CXR-2, InferRead, Xvision, and ChestEye (AUCs 0·8–0·9). XrayAME, RADIFY, and TiSepX-TB had AUC under 0·8. Thresholds varied notably across these products and different versions of the same products. Certain products (Lunit, Nexus, JF CXR-2, and qXR) maintained high sensitivity (>90%) across a wide threshold range while reducing the number of individuals requiring confirmatory diagnostic testing. All products generally performed worst in older individuals, people with previous tuberculosis, and people with HIV. Variations in thresholds, sensitivity, and specificity existed across groups and settings.</p></div><div><h3>Interpretation</h3><p>Several previously unevaluated products performed similarly to those evaluated by WHO. Thresholds differed across products and demographic subgroups. The rapid emergence of products and versions necessitates a global strategy to validate new versions and software to support CAD product and threshold selections.</p></div><div><h3>Funding</h3><p>Government of Canada.</p></div>","PeriodicalId":48534,"journal":{"name":"Lancet Digital Health","volume":null,"pages":null},"PeriodicalIF":23.8,"publicationDate":"2024-07-19","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.sciencedirect.com/science/article/pii/S2589750024001183/pdfft?md5=775b5ed834f92e1ac27b79991982d09e&pid=1-s2.0-S2589750024001183-main.pdf","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141735366","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
扫码关注我们
求助内容：
应助结果提醒方式：
京公网安备 11010802042870号