Automated structured data extraction from intraoperative echocardiography reports using large language models

IF 9.2 1区医学 Q1 ANESTHESIOLOGY British journal of anaesthesia Pub Date : 2025-05-01 Epub Date: 2025-03-03 DOI:10.1016/j.bja.2025.01.028

Emily J. MacKay , Shir Goldfinger , Trevor J. Chan , Rachel H. Grasfield , Vikram J. Eswar , Kelly Li , Quy Cao , Alison M. Pouch

{"title":"Automated structured data extraction from intraoperative echocardiography reports using large language models","authors":"Emily J. MacKay , Shir Goldfinger , Trevor J. Chan , Rachel H. Grasfield , Vikram J. Eswar , Kelly Li , Quy Cao , Alison M. Pouch","doi":"10.1016/j.bja.2025.01.028","DOIUrl":null,"url":null,"abstract":"<div><h3>Background</h3><div>Consensus-based large language model (LLM) ensembles might provide an automated solution for extracting structured data from unstructured text in echocardiography reports.</div></div><div><h3>Methods</h3><div>This cross-sectional study utilised 600 intraoperative transoesophageal reports (100 for prompt engineering; 500 for testing) randomly sampled from 7106 adult patients undergoing cardiac surgery at two hospitals within the University of Pennsylvania Healthcare System. Three echocardiographic parameters (left ventricular ejection fraction, right ventricular systolic function, and tricuspid regurgitation) were extracted from both the presurgical and postsurgical sections of the reports. LLM ensembles were generated using five open-source LLMs and four voting strategies: (1) unanimous (five out of five in agreement); (2) supermajority (four or more of five in agreement); (3) majority (three or more of five in agreement); and (4) plurality (two or more of five in agreement). Returned LLM ensemble responses were compared with the reference standard dataset to calculate raw accuracy, consensus accuracy, error rate, and yield.</div></div><div><h3>Results</h3><div>Of the four LLM ensembles, the unanimous LLM ensemble achieved the highest consensus accuracies (99.4% presurgical; 97.9% postsurgical) and the lowest error rates (0.6% presurgical; 2.1% postsurgical) but had the lowest data extraction yields (81.7% presurgical; 80.5% postsurgical) and the lowest raw accuracies (81.2% presurgical; 78.9% postsurgical). In contrast, the plurality LLM ensemble achieved the highest raw accuracies (96.1% presurgical; 93.7% postsurgical) and the highest data extraction yields (99.4% presurgical; 98.9% postsurgical) but had the lowest consensus accuracies (96.7% presurgical; 94.7% postsurgical) and highest error rates (3.3% presurgical; 5.3% postsurgical).</div></div><div><h3>Conclusions</h3><div>A consensus-based LLM ensemble successfully generated structured data from unstructured text contained in intraoperative transoesophageal reports.</div></div>","PeriodicalId":9250,"journal":{"name":"British journal of anaesthesia","volume":"134 5","pages":"Pages 1308-1317"},"PeriodicalIF":9.2000,"publicationDate":"2025-05-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"British journal of anaesthesia","FirstCategoryId":"3","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S0007091225000650","RegionNum":1,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"2025/3/3 0:00:00","PubModel":"Epub","JCR":"Q1","JCRName":"ANESTHESIOLOGY","Score":null,"Total":0}

引用次数: 0

Abstract

Background

Consensus-based large language model (LLM) ensembles might provide an automated solution for extracting structured data from unstructured text in echocardiography reports.

Methods

This cross-sectional study utilised 600 intraoperative transoesophageal reports (100 for prompt engineering; 500 for testing) randomly sampled from 7106 adult patients undergoing cardiac surgery at two hospitals within the University of Pennsylvania Healthcare System. Three echocardiographic parameters (left ventricular ejection fraction, right ventricular systolic function, and tricuspid regurgitation) were extracted from both the presurgical and postsurgical sections of the reports. LLM ensembles were generated using five open-source LLMs and four voting strategies: (1) unanimous (five out of five in agreement); (2) supermajority (four or more of five in agreement); (3) majority (three or more of five in agreement); and (4) plurality (two or more of five in agreement). Returned LLM ensemble responses were compared with the reference standard dataset to calculate raw accuracy, consensus accuracy, error rate, and yield.

Results

Of the four LLM ensembles, the unanimous LLM ensemble achieved the highest consensus accuracies (99.4% presurgical; 97.9% postsurgical) and the lowest error rates (0.6% presurgical; 2.1% postsurgical) but had the lowest data extraction yields (81.7% presurgical; 80.5% postsurgical) and the lowest raw accuracies (81.2% presurgical; 78.9% postsurgical). In contrast, the plurality LLM ensemble achieved the highest raw accuracies (96.1% presurgical; 93.7% postsurgical) and the highest data extraction yields (99.4% presurgical; 98.9% postsurgical) but had the lowest consensus accuracies (96.7% presurgical; 94.7% postsurgical) and highest error rates (3.3% presurgical; 5.3% postsurgical).

Conclusions

A consensus-based LLM ensemble successfully generated structured data from unstructured text contained in intraoperative transoesophageal reports.

查看原文

微信好友朋友圈 QQ好友复制链接

本刊更多论文

使用大型语言模型从术中超声心动图报告中自动提取结构化数据。

背景：基于共识的大语言模型（LLM）集成可能为从超声心动图报告中的非结构化文本中提取结构化数据提供自动化解决方案。方法：这项横断面研究使用了600例术中经食管报告(100例为及时工程；从宾夕法尼亚大学医疗保健系统内两家医院接受心脏手术的7106名成年患者中随机抽取500名用于测试。三个超声心动图参数（左心室射血分数，右心室收缩功能和三尖瓣反流）从术前和术后切片中提取。使用五个开源LLM和四种投票策略生成LLM集合：(1)一致（五分之五同意）；(2)绝对多数（同意五人中的四人以上）；（三）多数（五人一致同意中的三人以上）；(4)复数（五个一致中的两个或两个以上）。将返回的LLM集成响应与参考标准数据集进行比较，以计算原始准确性、一致性准确性、错误率和产率。结果：在四种LLM组合中，一致LLM组合的一致性准确性最高(术前99.4%；97.9%为术后)，错误率最低(术前0.6%；（2.1%），但数据提取率最低(术前81.7%；80.5%术后)，最低的原始准确率(81.2%术前；手术后的78.9%)。相比之下，多个LLM集合获得了最高的原始准确率(术前96.1%；93.7%为术后)，数据提取率最高(术前99.4%；98.9%)，但一致性准确率最低(术前96.7%；术后94.7%)，错误率最高(术前3.3%；手术后的5.3%)。结论：基于共识的LLM集成成功地从术中经食管报告中包含的非结构化文本生成结构化数据。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文去求助

来源期刊

British journal of anaesthesia 医学-麻醉学

CiteScore

13.50

自引率

7.10%

发文量

488

审稿时长

27 days

期刊介绍： The British Journal of Anaesthesia (BJA) is a prestigious publication that covers a wide range of topics in anaesthesia, critical care medicine, pain medicine, and perioperative medicine. It aims to disseminate high-impact original research, spanning fundamental, translational, and clinical sciences, as well as clinical practice, technology, education, and training. Additionally, the journal features review articles, notable case reports, correspondence, and special articles that appeal to a broader audience. The BJA is proudly associated with The Royal College of Anaesthetists, The College of Anaesthesiologists of Ireland, and The Hong Kong College of Anaesthesiologists. This partnership provides members of these esteemed institutions with access to not only the BJA but also its sister publication, BJA Education. It is essential to note that both journals maintain their editorial independence. Overall, the BJA offers a diverse and comprehensive platform for anaesthetists, critical care physicians, pain specialists, and perioperative medicine practitioners to contribute and stay updated with the latest advancements in their respective fields.