International Journal of Medical Informatics最新文献

Tracking provenance in clinical data warehouses for quality management 跟踪临床数据仓库中的出处，促进质量管理

IF 3.7 2区医学 Q2 COMPUTER SCIENCE, INFORMATION SYSTEMS

International Journal of Medical Informatics

Pub Date : 2024-11-02 DOI: 10.1016/j.ijmedinf.2024.105690

Introduction

Data provenance, which documents the origin, history, and transformations of data, can enhance the reproducibility of processing workflows and help to address errors and quality issues. In this work, we focus on tracking and utilizing provenance information as part of quality management in Extract-Transform-Load (ETL) processes used to build clinical data warehouses.

Methods

We designed and implemented a framework that automatically tracks how data flows through an ETL process and detects errors and quality problems during processing. This information is then reported against an Application Programming Interface (API) that stores the issues along with contextual information on their location within the data being transformed and the overall workflow. We further designed a dashboard that supports health data engineers with inspecting the encountered issues and tracing them back to their root causes.

Results

The framework was implemented in Java using the Spring Framework and integrated into ETL processes for Informatics for Integrating Biology and the Bedside (i2b2). The dashboard was realized using Grafana. We evaluated our approach on three different ETL processes for real-world datasets used to integrate them into our i2b2 clinical data warehouse. Using the provenance dashboard, we were able to identify frequent error patterns and link them to specific data points from the sources as well as ETL process steps. Provenance tracking increased the execution times of loading processes with an impact depending on the number of identified issues.

Conclusions

Provenance tracking can be a valuable tool for implementing continuous quality management for ETL processes. Relevant information can be collected from existing ETL workloads using dedicated APIs and visualized through dashboards, which support the identification of frequent patterns of problems together with their root causes, providing valuable information for improvements.

导言数据出处记录了数据的来源、历史和转换，可以提高处理工作流程的可重复性，并有助于解决错误和质量问题。我们设计并实施了一个框架，该框架可自动跟踪数据如何在 ETL 流程中流动，并检测处理过程中的错误和质量问题。然后根据应用程序接口（API）报告这些信息，应用程序接口会存储这些问题以及它们在正在转换的数据中的位置和整个工作流程的上下文信息。我们还设计了一个仪表盘，支持健康数据工程师检查遇到的问题并追溯其根源。结果该框架使用 Spring 框架在 Java 中实现，并集成到了生物与床边整合信息学（i2b2）的 ETL 流程中。仪表盘使用 Grafana 实现。我们在三个不同的 ETL 流程中对我们的方法进行了评估，这些流程用于将真实世界的数据集集成到我们的 i2b2 临床数据仓库中。通过使用出处仪表板，我们能够识别出经常出现的错误模式，并将它们与数据源中的特定数据点以及 ETL 流程步骤联系起来。出处跟踪增加了加载流程的执行时间，其影响取决于已识别问题的数量。可以使用专用的 API 从现有的 ETL 工作负载中收集相关信息，并通过仪表盘将其可视化，从而支持识别经常出现的问题模式及其根本原因，为改进工作提供有价值的信息。

{"title":"Tracking provenance in clinical data warehouses for quality management","authors":"","doi":"10.1016/j.ijmedinf.2024.105690","DOIUrl":"10.1016/j.ijmedinf.2024.105690","url":null,"abstract":"<div><h3>Introduction</h3><div>Data provenance, which documents the origin, history, and transformations of data, can enhance the reproducibility of processing workflows and help to address errors and quality issues. In this work, we focus on tracking and utilizing provenance information as part of quality management in Extract-Transform-Load (ETL) processes used to build clinical data warehouses.</div></div><div><h3>Methods</h3><div>We designed and implemented a framework that automatically tracks how data flows through an ETL process and detects errors and quality problems during processing. This information is then reported against an Application Programming Interface (API) that stores the issues along with contextual information on their location within the data being transformed and the overall workflow. We further designed a dashboard that supports health data engineers with inspecting the encountered issues and tracing them back to their root causes.</div></div><div><h3>Results</h3><div>The framework was implemented in Java using the Spring Framework and integrated into ETL processes for Informatics for Integrating Biology and the Bedside (i2b2). The dashboard was realized using Grafana. We evaluated our approach on three different ETL processes for real-world datasets used to integrate them into our i2b2 clinical data warehouse. Using the provenance dashboard, we were able to identify frequent error patterns and link them to specific data points from the sources as well as ETL process steps. Provenance tracking increased the execution times of loading processes with an impact depending on the number of identified issues.</div></div><div><h3>Conclusions</h3><div>Provenance tracking can be a valuable tool for implementing continuous quality management for ETL processes. Relevant information can be collected from existing ETL workloads using dedicated APIs and visualized through dashboards, which support the identification of frequent patterns of problems together with their root causes, providing valuable information for improvements.</div></div>","PeriodicalId":54950,"journal":{"name":"International Journal of Medical Informatics","volume":null,"pages":null},"PeriodicalIF":3.7,"publicationDate":"2024-11-02","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142586204","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Application of the openEHR reference model for PGHD: A case study on the DH-Convener initiative 将开放式电子病历参考模型应用于 PGHD：关于 DH-Convener 计划的案例研究

IF 3.7 2区医学 Q2 COMPUTER SCIENCE, INFORMATION SYSTEMS

International Journal of Medical Informatics

Pub Date : 2024-11-02 DOI: 10.1016/j.ijmedinf.2024.105686

Objectives

Patient-Generated Health Data (PGHD) is increasingly influential in therapy and diagnostic decisions. PGHD should be integrated into electronic health records (EHR) to maximize its utility. This study evaluates the openEHR Reference Model (RM) compatibility with the DH-Convener initiative’s modules (Data Collection Module and Data Connector Module) as a potential concept for standardizing PGHD across wearable health devices, focusing on achieving interoperability.

Materials and Methods

The study analyzes various types of PGHD, assessing the data formats and structures used by wearable tools. We evaluate openEHR RM specification with our initiative, DH-Convenor, focusing on PGHD semantic interoperability challenges. We evaluated current Archetypes and Templates that are now created and exist on openEHR Clinical Knowledge Management (CKM) and mapped them to our requirements. The DH-Convener modules are examined for their compatibility in standardizing PGHD integration into openEHR clinical workflows and compared with other existing standards for flexibility, scalability, and interoperability.

Results

The findings indicate that the diversity in data formats across wearable tools and openEHR shows strong potential as unifying data models based on the DH-Convener’s modules. It supports a wide range of PGHD types in existing archetypes and aligns well with our initiative’s requirements for storing PGHD, enabling more seamless integration into EHR systems.

Discussion

Integrating PGHD into EHR is crucial for personalized healthcare, but inconsistent device formats hinder interoperability. The DH-Convener leverages openEHR to provide a strong solution, though stakeholder collaboration remains essential. Our initiative demonstrates openEHR’s ability to ensure consistency, particularly in Europe.

Conclusion

We aligned the openEHR layers with the DH-Convener modules, demonstrating openEHR’s compatibility for storing PGHD and supporting interoperability goals, such as standardized storage and seamless data transfer to Austria’s national EHR. Future efforts should prioritize promoting these models and ensuring their adaptability to emerging wearable devices.

目的患者生成的健康数据 (PGHD) 对治疗和诊断决策的影响越来越大。应将 PGHD 整合到电子健康记录 (EHR) 中，以最大限度地发挥其效用。本研究评估了 openEHR 参考模型 (RM) 与 DH-Convener 计划模块（数据收集模块和数据连接器模块）的兼容性，将其作为实现可穿戴健康设备 PGHD 标准化的潜在概念，重点关注实现互操作性。我们评估了 openEHR RM 规范和我们的倡议 DH-Convenor，重点关注 PGHD 语义互操作性方面的挑战。我们评估了目前在 openEHR 临床知识管理 (CKM) 上创建和存在的原型和模板，并将其与我们的要求进行了映射。我们检查了 DH-Convener 模块在将 PGHD 标准化集成到 openEHR 临床工作流中的兼容性，并就灵活性、可扩展性和互操作性与其他现有标准进行了比较。结果研究结果表明，可穿戴工具和 openEHR 数据格式的多样性显示出基于 DH-Convener 模块的统一数据模型的强大潜力。它支持现有原型中广泛的 PGHD 类型，并与我们的倡议对存储 PGHD 的要求非常吻合，从而能够更无缝地集成到 EHR 系统中。讨论将 PGHD 集成到 EHR 中对个性化医疗保健至关重要，但不一致的设备格式阻碍了互操作性。DH-Convener 利用 openEHR 提供了一个强大的解决方案，但利益相关者的合作仍然至关重要。结论我们将 openEHR 层与 DH-Convener 模块进行了统一，证明 openEHR 在存储 PGHD 和支持互操作性目标（如标准化存储和将数据无缝传输到奥地利国家 EHR）方面具有兼容性。未来的工作应优先考虑推广这些模式，并确保它们能适应新兴的可穿戴设备。

{"title":"Application of the openEHR reference model for PGHD: A case study on the DH-Convener initiative","authors":"","doi":"10.1016/j.ijmedinf.2024.105686","DOIUrl":"10.1016/j.ijmedinf.2024.105686","url":null,"abstract":"<div><h3>Objectives</h3><div>Patient-Generated Health Data (PGHD) is increasingly influential in therapy and diagnostic decisions. PGHD should be integrated into electronic health records (EHR) to maximize its utility. This study evaluates the openEHR Reference Model (RM) compatibility with the DH-Convener initiative’s modules (Data Collection Module and Data Connector Module) as a potential concept for standardizing PGHD across wearable health devices, focusing on achieving interoperability.</div></div><div><h3>Materials and Methods</h3><div>The study analyzes various types of PGHD, assessing the data formats and structures used by wearable tools. We evaluate openEHR RM specification with our initiative, DH-Convenor, focusing on PGHD semantic interoperability challenges. We evaluated current Archetypes and Templates that are now created and exist on openEHR Clinical Knowledge Management (CKM) and mapped them to our requirements. The DH-Convener modules are examined for their compatibility in standardizing PGHD integration into openEHR clinical workflows and compared with other existing standards for flexibility, scalability, and interoperability.</div></div><div><h3>Results</h3><div>The findings indicate that the diversity in data formats across wearable tools and openEHR shows strong potential as unifying data models based on the DH-Convener’s modules. It supports a wide range of PGHD types in existing archetypes and aligns well with our initiative’s requirements for storing PGHD, enabling more seamless integration into EHR systems.</div></div><div><h3>Discussion</h3><div>Integrating PGHD into EHR is crucial for personalized healthcare, but inconsistent device formats hinder interoperability. The DH-Convener leverages openEHR to provide a strong solution, though stakeholder collaboration remains essential. Our initiative demonstrates openEHR’s ability to ensure consistency, particularly in Europe.</div></div><div><h3>Conclusion</h3><div>We aligned the openEHR layers with the DH-Convener modules, demonstrating openEHR’s compatibility for storing PGHD and supporting interoperability goals, such as standardized storage and seamless data transfer to Austria’s national EHR. Future efforts should prioritize promoting these models and ensuring their adaptability to emerging wearable devices.</div></div>","PeriodicalId":54950,"journal":{"name":"International Journal of Medical Informatics","volume":null,"pages":null},"PeriodicalIF":3.7,"publicationDate":"2024-11-02","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142586203","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Acute myocardial infarction risk prediction in emergency chest pain patients: An external validation study 急诊胸痛患者急性心肌梗死风险预测：外部验证研究

IF 3.7 2区医学 Q2 COMPUTER SCIENCE, INFORMATION SYSTEMS

International Journal of Medical Informatics

Pub Date : 2024-11-01 DOI: 10.1016/j.ijmedinf.2024.105683

Background

Chest pain is a common symptom that presents to the emergency department (ED), and its causes range from minor illnesses to serious diseases such as acute coronary syndrome. Accurate and timely diagnosis is essential for the efficient management and treatment of these patients.

Objective

This study aims to expand on a model previously developed by the Chi Mei Medical Group (CMMG) Emergency Department in 2020 to predict adverse cardiac events in patients with chest pain. The main goal is to evaluate the accuracy and generalizability of the model through external validation using data from other hospitals.

Methods

The initial model for this study was developed using data from three CMMG-affiliated hospitals in southern Taiwan. We utilized four supervised machine learning algorithms, namely random forest, logistic regression, support-vector clustering, and K-nearest neighbor, to predict the risk of acute myocardial infarction within a one month for emergency chest pain patients. The study used the model with the best area under the curve (AUC), recall and precision for external validation. The external validated data source was data collected from three hospitals associated with Taipei Medical University (TMU) in northern Taiwan. Results: The original best model constructed by CMMG exhibited an AUC of 0.822, an accuracy of 0.740, a recall of 0.741, a precision of 0.566, a specificity of 0.740, and an NPV of 0.861. Subsequently, during the external validation phase, CMMG’s top-performing model demonstrated acceptable validation result with TMU’s data, achieving an AUC of 0.63, an accuracy of 0.661, a recall of 0.593, a precision of 0.243, a specificity of 0.691, and an NPV of 0.900. While the results indicate that the model’s performance varied across different datasets and are not outstanding, the model is still acceptable for clinical application as a preliminary decision-support tool.

Conclusion

This study highlights the importance of external validation to confirm the applicability of the previously developed predictive model in other hospital settings. Although the model shows potential in assessing chest pain patients in the ED, its broad clinical application requires further validation to ensure it can improve patient outcomes and optimize healthcare resource allocation.

背景胸痛是急诊科（ED）常见的症状，其原因从轻微疾病到严重疾病（如急性冠状动脉综合征）不等。本研究旨在扩展奇美医疗集团（CMMG）急诊科于 2020 年开发的预测胸痛患者不良心脏事件的模型。主要目的是通过使用其他医院的数据进行外部验证，评估模型的准确性和可推广性。方法本研究的初始模型是使用奇美医疗集团在台湾南部的三家附属医院的数据开发的。我们使用了四种有监督的机器学习算法，即随机森林、逻辑回归、支持向量聚类和 K 最近邻，来预测急诊胸痛患者在一个月内发生急性心肌梗死的风险。研究采用了曲线下面积（AUC）、召回率和精确度最佳的模型进行外部验证。外部验证数据来源于台湾北部台北医学大学的三家附属医院。结果由 CMMG 构建的原始最佳模型的 AUC 为 0.822，准确度为 0.740，召回率为 0.741，精确度为 0.566，特异性为 0.740，净现值为 0.861。随后，在外部验证阶段，CMMG 表现最出色的模型通过屯门大学的数据获得了可接受的验证结果，AUC 为 0.63，准确率为 0.661，召回率为 0.593，精确度为 0.243，特异性为 0.691，净现值为 0.900。虽然结果表明该模型在不同数据集上的表现各不相同，并不突出，但该模型作为初步的决策支持工具仍可用于临床应用。虽然该模型在评估急诊室胸痛患者方面显示出了潜力，但其广泛的临床应用还需要进一步验证，以确保它能改善患者预后并优化医疗资源分配。

{"title":"Acute myocardial infarction risk prediction in emergency chest pain patients: An external validation study","authors":"","doi":"10.1016/j.ijmedinf.2024.105683","DOIUrl":"10.1016/j.ijmedinf.2024.105683","url":null,"abstract":"<div><h3>Background</h3><div>Chest pain is a common symptom that presents to the emergency department (ED), and its causes range from minor illnesses to serious diseases such as acute coronary syndrome. Accurate and timely diagnosis is essential for the efficient management and treatment of these patients.</div></div><div><h3>Objective</h3><div>This study aims to expand on a model previously developed by the Chi Mei Medical Group (CMMG) Emergency Department in 2020 to predict adverse cardiac events in patients with chest pain. The main goal is to evaluate the accuracy and generalizability of the model through external validation using data from other hospitals.</div></div><div><h3>Methods</h3><div>The initial model for this study was developed using data from three CMMG-affiliated hospitals in southern Taiwan. We utilized four supervised machine learning algorithms, namely random forest, logistic regression, support-vector clustering, and K-nearest neighbor, to predict the risk of acute myocardial infarction within a one month for emergency chest pain patients. The study used the model with the best area under the curve (AUC), recall and precision for external validation. The external validated data source was data collected from three hospitals associated with Taipei Medical University (TMU) in northern Taiwan. <strong>Results:</strong> The original best model constructed by CMMG exhibited an AUC of 0.822, an accuracy of 0.740, a recall of 0.741, a precision of 0.566, a specificity of 0.740, and an NPV of 0.861. Subsequently, during the external validation phase, CMMG’s top-performing model demonstrated acceptable validation result with TMU’s data, achieving an AUC of 0.63, an accuracy of 0.661, a recall of 0.593, a precision of 0.243, a specificity of 0.691, and an NPV of 0.900. While the results indicate that the model’s performance varied across different datasets and are not outstanding, the model is still acceptable for clinical application as a preliminary decision-support tool.</div></div><div><h3>Conclusion</h3><div>This study highlights the importance of external validation to confirm the applicability of the previously developed predictive model in other hospital settings. Although the model shows potential in assessing chest pain patients in the ED, its broad clinical application requires further validation to ensure it can improve patient outcomes and optimize healthcare resource allocation.</div></div>","PeriodicalId":54950,"journal":{"name":"International Journal of Medical Informatics","volume":null,"pages":null},"PeriodicalIF":3.7,"publicationDate":"2024-11-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142586728","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Healthcare professionals’ cross-organizational access to electronic health records: A scoping review 医疗保健专业人员跨组织访问电子健康记录：范围界定审查

IF 3.7 2区医学 Q2 COMPUTER SCIENCE, INFORMATION SYSTEMS

International Journal of Medical Informatics

Pub Date : 2024-11-01 DOI: 10.1016/j.ijmedinf.2024.105688

Background

Cross-organizational access to shared electronic health records can enhance integrated, people-centered health services. However, a gap remains between these potential benefits and the limited support currently offered by electronic health records. The Valkyrie research project aims to bridge this gap by developing a technical prototype of an architecture to promote healthcare service coordination.

Objective

To inform the Valkyrie project, we aimed to evaluate approaches for healthcare professionals’ access to electronic health records across healthcare providers and identify factors influencing the success and failure of these approaches.

Materials and methods

Using the Joanna Briggs Institute guidance for scoping reviews, searches were conducted in six research databases and grey literature, without limitations on year or language. Papers selected for full-text review were analyzed, and data was extracted using standardized forms that reflected the population, concept, and context framework and the categorization model used in the qualitative analysis of the barriers and facilitators reported in the included papers.

Results

Among the 290 identified papers, five were deemed eligible for full-text review. The included papers were heterogeneous in country, year of publication, study setting, implementation level, and access approaches to electronic health records, highlighting various techniques, from federated to centralized, for accessing shared electronic health records.

Discussion and conclusion

The review did not identify one single superior access approach. However, a hybrid approach incorporating components from the different approaches combined with emerging technologies may benefit the Valkyrie project. The key facilitators were identified as improved information quality and flexible and easy access. In contrast, lack of trust and poor information quality were significant barriers to successful cross-organizational access to electronic health records. Future research should explore alternative access approaches, considering information quality, user training, and collegial trust across healthcare providers.

背景跨机构访问共享电子健康记录可以加强以人为本的综合医疗服务。然而，这些潜在的好处与电子健康记录目前提供的有限支持之间仍存在差距。为了给 Valkyrie 项目提供信息，我们旨在评估医疗保健专业人员跨医疗保健提供方访问电子健康记录的方法，并确定影响这些方法成功与失败的因素。对选中进行全文审阅的论文进行了分析，并使用标准化表格提取数据，这些表格反映了人口、概念和背景框架，以及对所收录论文中报告的障碍和促进因素进行定性分析时使用的分类模型。结果在所确定的 290 篇论文中，有 5 篇被认为符合全文审阅的条件。被收录的论文在国家、发表年份、研究背景、实施水平和电子病历访问方法方面各不相同，突出了从联合到集中等各种访问共享电子病历的技术。不过，将不同方法的组成部分与新兴技术相结合的混合方法可能会使瓦尔基里项目受益。关键的促进因素被认为是信息质量的提高和灵活便捷的访问。相比之下，缺乏信任和信息质量差是成功跨组织访问电子病历的主要障碍。未来的研究应考虑信息质量、用户培训和医疗服务提供者之间的同事信任，探索其他访问方法。

{"title":"Healthcare professionals’ cross-organizational access to electronic health records: A scoping review","authors":"","doi":"10.1016/j.ijmedinf.2024.105688","DOIUrl":"10.1016/j.ijmedinf.2024.105688","url":null,"abstract":"<div><h3>Background</h3><div>Cross-organizational access to shared electronic health records can enhance integrated, people-centered health services. However, a gap remains between these potential benefits and the limited support currently offered by electronic health records. The Valkyrie research project aims to bridge this gap by developing a technical prototype of an architecture to promote healthcare service coordination.</div></div><div><h3>Objective</h3><div>To inform the Valkyrie project, we aimed to evaluate approaches for healthcare professionals’ access to electronic health records across healthcare providers and identify factors influencing the success and failure of these approaches.</div></div><div><h3>Materials and methods</h3><div>Using the Joanna Briggs Institute guidance for scoping reviews, searches were conducted in six research databases and grey literature, without limitations on year or language. Papers selected for full-text review were analyzed, and data was extracted using standardized forms that reflected the population, concept, and context framework and the categorization model used in the qualitative analysis of the barriers and facilitators reported in the included papers.</div></div><div><h3>Results</h3><div>Among the 290 identified papers, five were deemed eligible for full-text review. The included papers were heterogeneous in country, year of publication, study setting, implementation level, and access approaches to electronic health records, highlighting various techniques, from federated to centralized, for accessing shared electronic health records.</div></div><div><h3>Discussion and conclusion</h3><div>The review did not identify one single superior access approach. However, a hybrid approach incorporating components from the different approaches combined with emerging technologies may benefit the Valkyrie project. The key facilitators were identified as improved information quality and flexible and easy access. In contrast, lack of trust and poor information quality were significant barriers to successful cross-organizational access to electronic health records. Future research should explore alternative access approaches, considering information quality, user training, and collegial trust across healthcare providers.</div></div>","PeriodicalId":54950,"journal":{"name":"International Journal of Medical Informatics","volume":null,"pages":null},"PeriodicalIF":3.7,"publicationDate":"2024-11-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142593198","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

IF 3.7 2区医学 Q2 COMPUTER SCIENCE, INFORMATION SYSTEMS

International Journal of Medical Informatics

Pub Date : 2024-10-31 DOI: 10.1016/j.ijmedinf.2024.105680

Objective

Electronic health record systems have made it possible for clinicians to use previously encountered similar cases to support clinical decision-making. However, most studies for similar case retrieval were based on single-modal data. The existing studies on cross-modal clinical case retrieval were limited. We aimed to develop a CRoss-Modal Retrieval (CRMR) model to retrieve similar clinical cases recorded in different data modalities.

Materials and methods

The publically available Medical Information Mart for Intensive Care-Chest X-ray (MIMIC-CXR) dataset was used for model development and testing. The CRMR model was designed as a modular model containing two feature extraction models, two feature transformation models, one feature transformation optimization model, and one case retrieval model. The ability to retrieve similar clinical cases recorded in different data modalities was facilitated by the use of contrastive deep learning and k-nearest neighbor search.

Results

The average retrieval precision, denoted as AP@k, of the developed CRMR model, were 76.9 %@5, 76.7 %@10, 76.5 %@20, 76.3 %@50, and 77.9 %@100, respectively. Here k is the number of similar cases returned after retrieval. The average retrieval time varied from 0.013 ms to 0.016 ms with k varying from 5 to 100. Moreover, the model can retrieve similar cases with the same multiple radiographic manifestations as the query case.

Discussion

The CRMR model has shown promising cross-modal retrieval performance in clinical case analysis, with the potential for future scalability and improvement in handling diverse disease types and data modalities. The CRMR model has promising potential to aid clinicians in making optimal and explainable clinical decisions.

目标电子健康记录系统使临床医生有可能利用以前遇到的类似病例来支持临床决策。然而，大多数关于类似病例检索的研究都是基于单一模式的数据。现有的跨模态临床病例检索研究非常有限。我们的目标是开发一个CRoss-Modal Retrieval（CRMR）模型，以检索不同数据模式下记录的类似临床病例。材料与方法公开可用的重症监护医学信息市场-胸部X光（MIMIC-CXR）数据集用于模型开发和测试。CRMR 模型被设计为一个模块化模型，包含两个特征提取模型、两个特征转换模型、一个特征转换优化模型和一个病例检索模型。结果所开发的 CRMR 模型的平均检索精度（以 AP@k 表示）分别为 76.9 %@5、76.7 %@10、76.5 %@20、76.3 %@50 和 77.9 %@100。这里 k 是检索后返回的相似案例数。当 k 为 5 到 100 时，平均检索时间从 0.013 毫秒到 0.016 毫秒不等。此外，该模型还能检索出与查询病例具有相同的多种放射学表现的相似病例。讨论 CRMR 模型在临床病例分析中显示出了良好的跨模态检索性能，在处理不同的疾病类型和数据模式方面具有可扩展性和改进潜力。CRMR 模型有望帮助临床医生做出最佳和可解释的临床决策。

{"title":"Cross-modal similar clinical case retrieval using a modular model based on contrastive learning and k-nearest neighbor search","authors":"","doi":"10.1016/j.ijmedinf.2024.105680","DOIUrl":"10.1016/j.ijmedinf.2024.105680","url":null,"abstract":"<div><h3>Objective</h3><div>Electronic health record systems have made it possible for clinicians to use previously encountered similar cases to support clinical decision-making. However, most studies for similar case retrieval were based on single-modal data. The existing studies on cross-modal clinical case retrieval were limited. We aimed to develop a CRoss-Modal Retrieval (CRMR) model to retrieve similar clinical cases recorded in different data modalities.</div></div><div><h3>Materials and methods</h3><div>The publically available Medical Information Mart for Intensive Care-Chest X-ray (MIMIC-CXR) dataset was used for model development and testing. The CRMR model was designed as a modular model containing two feature extraction models, two feature transformation models, one feature transformation optimization model, and one case retrieval model. The ability to retrieve similar clinical cases recorded in different data modalities was facilitated by the use of contrastive deep learning and <em>k</em>-nearest neighbor search.</div></div><div><h3>Results</h3><div>The average retrieval precision, denoted as AP@<em>k</em>, of the developed CRMR model, were 76.9 %@5, 76.7 %@10, 76.5 %@20, 76.3 %@50, and 77.9 %@100, respectively. Here <em>k</em> is the number of similar cases returned after retrieval. The average retrieval time varied from 0.013 ms to 0.016 ms with <em>k</em> varying from 5 to 100. Moreover, the model can retrieve similar cases with the same multiple radiographic manifestations as the query case.</div></div><div><h3>Discussion</h3><div>The CRMR model has shown promising cross-modal retrieval performance in clinical case analysis, with the potential for future scalability and improvement in handling diverse disease types and data modalities. The CRMR model has promising potential to aid clinicians in making optimal and explainable clinical decisions.</div></div>","PeriodicalId":54950,"journal":{"name":"International Journal of Medical Informatics","volume":null,"pages":null},"PeriodicalIF":3.7,"publicationDate":"2024-10-31","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142579092","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Expert opinion elicitation for assisting deep learning based Lyme disease classifier with patient data 征询专家意见，利用患者数据辅助基于深度学习的莱姆病分类器

IF 3.7 2区医学 Q2 COMPUTER SCIENCE, INFORMATION SYSTEMS

International Journal of Medical Informatics

Pub Date : 2024-10-31 DOI: 10.1016/j.ijmedinf.2024.105682

Background

Diagnosing erythema migrans (EM) skin lesion, the most common early symptom of Lyme disease, using deep learning techniques can be effective to prevent long-term complications. Existing works on deep learning based EM recognition only utilizes lesion image due to the lack of a dataset of Lyme disease related images with associated patient data. Doctors rely on patient information about the background of the skin lesion to confirm their diagnosis. To assist deep learning model with a probability score calculated from patient data, this study elicited opinions from fifteen expert doctors. To the best of our knowledge, this is the first expert elicitation work to calculate Lyme disease probability from patient data.

Methods

For the elicitation process, a questionnaire with questions and possible answers related to EM was prepared. Doctors provided relative weights to different answers to the questions. We converted doctors' evaluations to probability scores using Gaussian mixture based density estimation. We exploited formal concept analysis and decision tree for elicited model validation and explanation. We also proposed an algorithm for combining independent probability estimates from multiple modalities, such as merging the EM probability score from a deep learning image classifier with the elicited score from patient data.

Results

We successfully elicited opinions from fifteen expert doctors to create a model for obtaining EM probability scores from patient data.

Conclusions

The elicited probability score and the proposed algorithm can be utilized to make image based deep learning Lyme disease pre-scanners robust. The proposed elicitation and validation process is easy for doctors to follow and can help address related medical diagnosis problems where it is challenging to collect patient data.

背景利用深度学习技术诊断莱姆病最常见的早期症状--迁延性红斑（EM）皮损，可以有效预防长期并发症。由于缺乏与莱姆病相关的图像数据集和相关患者数据，现有基于深度学习的 EM 识别工作只能利用皮损图像。医生只能依靠患者提供的皮损背景信息来确诊。为了帮助深度学习模型从患者数据中计算出概率分数，本研究征求了 15 位专家医生的意见。据我们所知，这是首次从患者数据中计算莱姆病概率的专家征询工作。方法在征询过程中，我们准备了一份问卷，其中包含与EM相关的问题和可能的答案。医生对问题的不同答案给出了相对权重。我们使用基于高斯混合物的密度估计法将医生的评价转换为概率分数。我们利用正式概念分析和决策树来验证和解释模型。我们还提出了一种算法，用于合并来自多种模式的独立概率估计值，例如将来自深度学习图像分类器的EM概率分数与来自患者数据的诱导分数合并。结果我们成功地从15位专家医生那里获得了意见，从而创建了一个从患者数据中获得EM概率分数的模型。所提出的诱导和验证过程对医生来说很容易操作，有助于解决收集患者数据具有挑战性的相关医疗诊断问题。

{"title":"Expert opinion elicitation for assisting deep learning based Lyme disease classifier with patient data","authors":"","doi":"10.1016/j.ijmedinf.2024.105682","DOIUrl":"10.1016/j.ijmedinf.2024.105682","url":null,"abstract":"<div><h3>Background</h3><div>Diagnosing erythema migrans (EM) skin lesion, the most common early symptom of Lyme disease, using deep learning techniques can be effective to prevent long-term complications. Existing works on deep learning based EM recognition only utilizes lesion image due to the lack of a dataset of Lyme disease related images with associated patient data. Doctors rely on patient information about the background of the skin lesion to confirm their diagnosis. To assist deep learning model with a probability score calculated from patient data, this study elicited opinions from fifteen expert doctors. To the best of our knowledge, this is the first expert elicitation work to calculate Lyme disease probability from patient data.</div></div><div><h3>Methods</h3><div>For the elicitation process, a questionnaire with questions and possible answers related to EM was prepared. Doctors provided relative weights to different answers to the questions. We converted doctors' evaluations to probability scores using Gaussian mixture based density estimation. We exploited formal concept analysis and decision tree for elicited model validation and explanation. We also proposed an algorithm for combining independent probability estimates from multiple modalities, such as merging the EM probability score from a deep learning image classifier with the elicited score from patient data.</div></div><div><h3>Results</h3><div>We successfully elicited opinions from fifteen expert doctors to create a model for obtaining EM probability scores from patient data.</div></div><div><h3>Conclusions</h3><div>The elicited probability score and the proposed algorithm can be utilized to make image based deep learning Lyme disease pre-scanners robust. The proposed elicitation and validation process is easy for doctors to follow and can help address related medical diagnosis problems where it is challenging to collect patient data.</div></div>","PeriodicalId":54950,"journal":{"name":"International Journal of Medical Informatics","volume":null,"pages":null},"PeriodicalIF":3.7,"publicationDate":"2024-10-31","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142586729","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

A literature-based approach to predict continuous hospital length of stay in adult acute care patients using admission variables: A single university center experience 基于文献的方法，利用入院变量预测成人急症患者的连续住院时间：一个大学中心的经验。

IF 3.7 2区医学 Q2 COMPUTER SCIENCE, INFORMATION SYSTEMS

International Journal of Medical Informatics

Pub Date : 2024-10-28 DOI: 10.1016/j.ijmedinf.2024.105678

Purpose

To review the existing literature on predicting length of stay (LOS) and to apply the findings on a Real World Data example in a single hospital.

Methods

Performing a literature review on PubMed and Embase, focusing on adults, acute conditions, and hospital-wide prediction of LOS, summarizing all the variables and statistical methods used to predict LOS. Then, we use this set of variables on a single university hospital and run an XGBoost model with Survival Cox regression on the LOS, as well as a logistic regression on binary LOS (cut-off at 4 days). Model metrics are the concordance index (c-index) and area under the curve (AUC).

Results

After applying the search strategy and exclusion criteria, 57 articles are included in the study. The list of variables is long, but mostly non-clinical data are used in the existing literature. A wide range of statistical methods are used, with a recent trend toward machine learning models. The XGBoost model results for the Cox regression in a C-index of 0.87, and the logistic regression on binary LOS has an AUC of 0.94.

Conclusions

Many variables identified in the literature are not available at the time of admission, yet they are still used in models for predicting LOS. Machine learning has become the preferred statistical approach in recent studies, though mainly for binary LOS predictions. Based on the current literature, it remains challenging to derive a practical and high performing model for continuous LOS prediction.

目的：回顾有关预测住院时间（LOS）的现有文献，并将研究结果应用于一家医院的真实世界数据示例：方法：在 PubMed 和 Embase 上进行文献综述，重点关注成人、急性病和全医院的 LOS 预测，总结用于预测 LOS 的所有变量和统计方法。然后，我们将这组变量用于一家大学医院，并运行一个 XGBoost 模型，对 LOS 进行生存 Cox 回归，并对二元 LOS（以 4 天为截止时间）进行逻辑回归。模型指标为一致性指数（c-index）和曲线下面积（AUC）：采用检索策略和排除标准后，本研究共纳入 57 篇文章。变量清单很长，但现有文献大多使用非临床数据。使用了多种统计方法，最近的趋势是使用机器学习模型。XGBoost 模型对 Cox 回归的 C 指数为 0.87，对二元 LOS 的逻辑回归 AUC 为 0.94：文献中确定的许多变量在入院时并不存在，但它们仍被用于预测 LOS 的模型中。在最近的研究中，机器学习已成为首选的统计方法，但主要用于二元 LOS 预测。从目前的文献来看，为连续 LOS 预测建立一个实用且高性能的模型仍具有挑战性。

{"title":"A literature-based approach to predict continuous hospital length of stay in adult acute care patients using admission variables: A single university center experience","authors":"","doi":"10.1016/j.ijmedinf.2024.105678","DOIUrl":"10.1016/j.ijmedinf.2024.105678","url":null,"abstract":"<div><h3>Purpose</h3><div>To review the existing literature on predicting length of stay (LOS) and to apply the findings on a Real World Data example in a single hospital.</div></div><div><h3>Methods</h3><div>Performing a literature review on PubMed and Embase, focusing on adults, acute conditions, and hospital-wide prediction of LOS, summarizing all the variables and statistical methods used to predict LOS. Then, we use this set of variables on a single university hospital and run an XGBoost model with Survival Cox regression on the LOS, as well as a logistic regression on binary LOS (cut-off at 4 days). Model metrics are the concordance index (c-index) and area under the curve (AUC).</div></div><div><h3>Results</h3><div>After applying the search strategy and exclusion criteria, 57 articles are included in the study. The list of variables is long, but mostly non-clinical data are used in the existing literature. A wide range of statistical methods are used, with a recent trend toward machine learning models. The XGBoost model results for the Cox regression in a C-index of 0.87, and the logistic regression on binary LOS has an AUC of 0.94.</div></div><div><h3>Conclusions</h3><div>Many variables identified in the literature are not available at the time of admission, yet they are still used in models for predicting LOS. Machine learning has become the preferred statistical approach in recent studies, though mainly for binary LOS predictions. Based on the current literature, it remains challenging to derive a practical and high performing model for continuous LOS prediction.</div></div>","PeriodicalId":54950,"journal":{"name":"International Journal of Medical Informatics","volume":null,"pages":null},"PeriodicalIF":3.7,"publicationDate":"2024-10-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142549025","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Post-Cardiac arrest outcome prediction using machine learning: A systematic review and meta-analysis 使用机器学习预测心脏骤停后的结果：系统回顾和荟萃分析

IF 3.7 2区医学 Q2 COMPUTER SCIENCE, INFORMATION SYSTEMS

International Journal of Medical Informatics

Pub Date : 2024-10-28 DOI: 10.1016/j.ijmedinf.2024.105659

Background

Early and reliable prognostication in post-cardiac arrest patients remains challenging, with various factors linked to return of spontaneous circulation (ROSC), survival, and neurological results. Machine learning and deep learning models show promise in improving these predictions. This systematic review and meta-analysis evaluates how effective these approaches are in predicting clinical outcomes at different time points using structured data.

Methods

This study followed PRISMA guidelines, involving a comprehensive search across PubMed, Scopus, and Web of Science databases until March 2024. Studies aimed at predicting ROSC, survival (or mortality), and neurological outcomes after cardiac arrest through the application of machine learning or deep learning techniques with structured data were included. Data extraction followed the guidelines of the CHARMS checklist, and the bias risk was evaluated using PROBAST tool. Models reporting the AUC metric with 95 % confidence intervals were incorporated into the quantitative synthesis and meta-analysis.

Results

After extracting 2,753 initial records, 41 studies met the inclusion criteria, yielding 97 machine learning and 16 deep learning models. The pooled AUC for predicting favorable neurological outcomes (CPC 1 or 2) at hospital discharge was 0.871 (95 % CI: 0.813 – 0.928) for machine learning models and 0.877 (95 % CI: 0.831–0.924) across deep learning algorithms. For survival prediction, this value was found to be 0.837 (95 % CI: 0.757–0.916). Considerable heterogeneity and high risk of bias were observed, mainly attributable to inadequate management of missing data and the absence of calibration plots. Most studies focused on pre-hospital factors, with age, sex, and initial arrest rhythm being the most frequent features.

Conclusion

Predictive models utilizing AI-based approaches, including machine and deep learning models exhibit enhanced effectiveness compared to previous regression algorithms, but significant heterogeneity and high risk of bias limit their dependability. Evaluating state-of-the-art deep learning models tailored for tabular data and their clinical generalizability can enhance outcome prediction after cardiac arrest.

背景心脏骤停后患者的早期可靠预后仍然具有挑战性，自发性循环恢复（ROSC）、存活率和神经功能结果与各种因素有关。机器学习和深度学习模型有望改善这些预测。本系统综述和荟萃分析评估了这些方法在使用结构化数据预测不同时间点的临床结果方面的有效性。方法本研究遵循 PRISMA 指南，在 2024 年 3 月之前对 PubMed、Scopus 和 Web of Science 数据库进行了全面检索。纳入的研究旨在通过应用机器学习或深度学习技术和结构化数据，预测心脏骤停后的ROSC、存活率（或死亡率）和神经系统预后。数据提取遵循CHARMS核对表指南，并使用PROBAST工具评估偏倚风险。结果在提取了2753条初始记录后，有41项研究符合纳入标准，产生了97个机器学习模型和16个深度学习模型。机器学习模型预测出院时良好神经功能预后（CPC 1 或 2）的集合 AUC 为 0.871（95 % CI：0.813 - 0.928），深度学习算法的集合 AUC 为 0.877（95 % CI：0.831-0.924）。在生存预测方面，这一数值为 0.837（95 % CI：0.757-0.916）。研究发现存在很大的异质性和较高的偏倚风险，这主要归因于对缺失数据的管理不足和校准图的缺失。结论与以往的回归算法相比，利用基于人工智能方法（包括机器学习和深度学习模型）的预测模型显示出更高的有效性，但显著的异质性和高偏倚风险限制了其可靠性。评估为表格数据定制的最先进的深度学习模型及其临床普适性可以提高心脏骤停后的预后预测。

{"title":"Post-Cardiac arrest outcome prediction using machine learning: A systematic review and meta-analysis","authors":"","doi":"10.1016/j.ijmedinf.2024.105659","DOIUrl":"10.1016/j.ijmedinf.2024.105659","url":null,"abstract":"<div><h3>Background</h3><div>Early and reliable prognostication in post-cardiac arrest patients remains challenging, with various factors linked to return of spontaneous circulation (ROSC), survival, and neurological results. Machine learning and deep learning models show promise in improving these predictions. This systematic review and <em>meta</em>-analysis evaluates how effective these approaches are in predicting clinical outcomes at different time points using structured data.</div></div><div><h3>Methods</h3><div>This study followed PRISMA guidelines, involving a comprehensive search across PubMed, Scopus, and Web of Science databases until March 2024. Studies aimed at predicting ROSC, survival (or mortality), and neurological outcomes after cardiac arrest through the application of machine learning or deep learning techniques with structured data were included. Data extraction followed the guidelines of the CHARMS checklist, and the bias risk was evaluated using PROBAST tool. Models reporting the AUC metric with 95 % confidence intervals were incorporated into the quantitative synthesis and <em>meta</em>-analysis.</div></div><div><h3>Results</h3><div>After extracting 2,753 initial records, 41 studies met the inclusion criteria, yielding 97 machine learning and 16 deep learning models. The pooled AUC for predicting favorable neurological outcomes (CPC 1 or 2) at hospital discharge was 0.871 (95 % CI: 0.813 – 0.928) for machine learning models and 0.877 (95 % CI: 0.831–0.924) across deep learning algorithms. For survival prediction, this value was found to be 0.837 (95 % CI: 0.757–0.916). Considerable heterogeneity and high risk of bias were observed, mainly attributable to inadequate management of missing data and the absence of calibration plots. Most studies focused on pre-hospital factors, with age, sex, and initial arrest rhythm being the most frequent features.</div></div><div><h3>Conclusion</h3><div>Predictive models utilizing AI-based approaches, including machine and deep learning models exhibit enhanced effectiveness compared to previous regression algorithms, but significant heterogeneity and high risk of bias limit their dependability. Evaluating state-of-the-art deep learning models tailored for tabular data and their clinical generalizability can enhance outcome prediction after cardiac arrest.</div></div>","PeriodicalId":54950,"journal":{"name":"International Journal of Medical Informatics","volume":null,"pages":null},"PeriodicalIF":3.7,"publicationDate":"2024-10-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142552267","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Evaluating the Effectiveness of advanced large language models in medical Knowledge: A Comparative study using Japanese national medical examination 评估高级大型语言模型在医学知识中的有效性：使用日本国家医学考试的比较研究。

IF 3.7 2区医学 Q2 COMPUTER SCIENCE, INFORMATION SYSTEMS

International Journal of Medical Informatics

Pub Date : 2024-10-28 DOI: 10.1016/j.ijmedinf.2024.105673

Study aims and objectives.

This study aims to evaluate the accuracy of medical knowledge in the most advanced LLMs (GPT-4o, GPT-4, Gemini 1.5 Pro, and Claude 3 Opus) as of 2024. It is the first to evaluate these LLMs using a non-English medical licensing exam. The insights from this study will guide educators, policymakers, and technical experts in the effective use of AI in medical education and clinical diagnosis.

Method

Authors inputted 790 questions from Japanese National Medical Examination into the chat windows of the LLMs to obtain responses. Two authors independently assessed the correctness. Authors analyzed the overall accuracy rates of the LLMs and compared their performance on image and non-image questions, questions of varying difficulty levels, general and clinical questions, and questions from different medical specialties. Additionally, authors examined the correlation between the number of publications and LLMs’ performance in different medical specialties.

Results

GPT-4o achieved highest accuracy rate of 89.2% and outperformed the other LLMs in overall performance and each specific category. All four LLMs performed better on non-image questions than image questions, with a 10% accuracy gap. They also performed better on easy questions compared to normal and difficult ones. GPT-4o achieved a 95.0% accuracy rate on easy questions, marking it as an effective knowledge source for medical education. Four LLMs performed worst on “Gastroenterology and Hepatology” specialty. There was a positive correlation between the number of publications and LLM performance in different specialties.

Conclusions

GPT-4o achieved an overall accuracy rate close to 90%, with 95.0% on easy questions, significantly outperforming the other LLMs. This indicates GPT-4o’s potential as a knowledge source for easy questions. Image-based questions and question difficulty significantly impact LLM accuracy. “Gastroenterology and Hepatology” is the specialty with the lowest performance. The LLMs’ performance across medical specialties correlates positively with the number of related publications.

研究目的和目标。本研究旨在评估截至 2024 年最先进的 LLM（GPT-4o、GPT-4、Gemini 1.5 Pro 和 Claude 3 Opus）中医学知识的准确性。这是首次使用非英语医学执照考试来评估这些 LLM。本研究的见解将指导教育工作者、政策制定者和技术专家在医学教育和临床诊断中有效使用人工智能：方法：作者将日本国家医学考试中的 790 个问题输入法学硕士的聊天窗口，以获取回复。两名作者独立评估正确率。作者分析了 LLMs 的总体正确率，并比较了它们在图像和非图像问题、不同难度的问题、普通和临床问题以及不同医学专业问题上的表现。此外，作者还研究了发表论文的数量与 LLMs 在不同医学专业中的表现之间的相关性：结果：GPT-4o 的准确率最高，达到 89.2%，在整体表现和每个特定类别中都优于其他 LLM。所有四种 LLM 在非图像问题上的表现均优于图像问题，准确率差距为 10%。它们在简单问题上的表现也优于普通问题和难题。GPT-4o 在简单问题上的准确率达到 95.0%，是医学教育的有效知识来源。四名法学硕士在 "胃肠病学和肝病学 "专业的成绩最差。在不同专业中，发表论文的数量与法学硕士的表现呈正相关：结论：GPT-4o 的总体准确率接近 90%，简单问题的准确率为 95.0%，明显优于其他 LLM。这表明 GPT-4o 具有作为简单问题知识源的潜力。基于图像的问题和问题难度对 LLM 的准确性有很大影响。"胃肠病学和肝病学 "是成绩最低的专业。LLM 在各医学专业中的表现与相关出版物的数量呈正相关。

{"title":"Evaluating the Effectiveness of advanced large language models in medical Knowledge: A Comparative study using Japanese national medical examination","authors":"","doi":"10.1016/j.ijmedinf.2024.105673","DOIUrl":"10.1016/j.ijmedinf.2024.105673","url":null,"abstract":"<div><div>Study aims and objectives.</div><div>This study aims to evaluate the accuracy of medical knowledge in the most advanced LLMs (GPT-4o, GPT-4, Gemini 1.5 Pro, and Claude 3 Opus) as of 2024. It is the first to evaluate these LLMs using a non-English medical licensing exam. The insights from this study will guide educators, policymakers, and technical experts in the effective use of AI in medical education and clinical diagnosis.</div></div><div><h3>Method</h3><div>Authors inputted 790 questions from Japanese National Medical Examination into the chat windows of the LLMs to obtain responses. Two authors independently assessed the correctness. Authors analyzed the overall accuracy rates of the LLMs and compared their performance on image and non-image questions, questions of varying difficulty levels, general and clinical questions, and questions from different medical specialties. Additionally, authors examined the correlation between the number of publications and LLMs’ performance in different medical specialties.</div></div><div><h3>Results</h3><div>GPT-4o achieved highest accuracy rate of 89.2% and outperformed the other LLMs in overall performance and each specific category. All four LLMs performed better on non-image questions than image questions, with a 10% accuracy gap. They also performed better on easy questions compared to normal and difficult ones. GPT-4o achieved a 95.0% accuracy rate on easy questions, marking it as an effective knowledge source for medical education. Four LLMs performed worst on “Gastroenterology and Hepatology” specialty. There was a positive correlation between the number of publications and LLM performance in different specialties.</div></div><div><h3>Conclusions</h3><div>GPT-4o achieved an overall accuracy rate close to 90%, with 95.0% on easy questions, significantly outperforming the other LLMs. This indicates GPT-4o’s potential as a knowledge source for easy questions. Image-based questions and question difficulty significantly impact LLM accuracy. “Gastroenterology and Hepatology” is the specialty with the lowest performance. The LLMs’ performance across medical specialties correlates positively with the number of related publications.</div></div>","PeriodicalId":54950,"journal":{"name":"International Journal of Medical Informatics","volume":null,"pages":null},"PeriodicalIF":3.7,"publicationDate":"2024-10-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142549026","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Enhancing real world data interoperability in healthcare: A methodological approach to laboratory unit harmonization 加强医疗保健领域真实世界数据的互操作性：实验室单位统一的方法论

IF 3.7 2区医学 Q2 COMPUTER SCIENCE, INFORMATION SYSTEMS

International Journal of Medical Informatics

Pub Date : 2024-10-28 DOI: 10.1016/j.ijmedinf.2024.105665

Objective

The primary aim of this study is to address the critical issue of non-standardized units in clinical laboratory data, which poses significant challenges to data interoperability and secondary usage. Despite UCUM (Unified Code for Units of Measure) offering a unique representation for laboratory test units, nearly 60% of laboratory codes in healthcare organizations use non-standard units. We sought to design, implement and test a methodology for the harmonization of units to the UCUM standards across a large research network.

Methods

Using dimensional analysis and a curated equivalence table, the proposed methodology harmonizes disparate units to UCUM standards. The process focused on identifying and converting non-UCUM conforming units, with the goal of enhancing data comparability and interoperability across different systems.

Results

The methodology successfully achieved over 90% coverage of laboratory data with units in UCUM standards across the TriNetX research network, a significant improvement from baseline measurements. This enhancement in unit standardization directly contributed to increased interoperability of laboratory data, facilitating more reliable and comparable data analysis across various healthcare organizations.

Conclusion

The successful harmonization of laboratory data units to UCUM standards represents a significant advancement in the field of biomedical informatics. By demonstrating a practical and effective approach to overcoming the challenges of non-standardized units, our study contributes to the broader efforts to improve data interoperability and usability for secondary purposes such as research and observational studies. Future work will focus on addressing the remaining gaps in unit standardization and exploring the implications of this methodology on clinical outcomes and research capabilities.

目的本研究的主要目的是解决临床实验室数据中的非标准化单位这一关键问题，因为它给数据互操作性和二次使用带来了巨大挑战。尽管 UCUM（计量单位统一代码）为实验室检验单位提供了独特的表示方法，但医疗机构中近 60% 的实验室代码使用的是非标准单位。我们试图在一个大型研究网络中设计、实施并测试一种将单位统一为 UCUM 标准的方法。方法利用维度分析和策划的等价表，建议的方法将不同的单位统一为 UCUM 标准。这一过程的重点是识别和转换不符合 UCUM 标准的单位，目的是提高不同系统间的数据可比性和互操作性。结果该方法成功地使 TriNetX 研究网络中符合 UCUM 标准的单位覆盖了 90% 以上的实验室数据，与基线测量值相比有了显著提高。这种单位标准化的提高直接促进了实验室数据互操作性的增强，为不同医疗机构之间进行更可靠、更可比的数据分析提供了便利。结论实验室数据单位与 UCUM 标准的成功统一是生物医学信息学领域的一大进步。通过展示克服非标准化单位挑战的实用有效方法，我们的研究为提高数据互操作性和二次用途（如研究和观察性研究）的可用性做出了更广泛的贡献。未来的工作将重点解决单位标准化方面的其余差距，并探索这种方法对临床结果和研究能力的影响。

{"title":"Enhancing real world data interoperability in healthcare: A methodological approach to laboratory unit harmonization","authors":"","doi":"10.1016/j.ijmedinf.2024.105665","DOIUrl":"10.1016/j.ijmedinf.2024.105665","url":null,"abstract":"<div><h3>Objective</h3><div>The primary aim of this study is to address the critical issue of non-standardized units in clinical laboratory data, which poses significant challenges to data interoperability and secondary usage. Despite UCUM (Unified Code for Units of Measure) offering a unique representation for laboratory test units, nearly 60% of laboratory codes in healthcare organizations use non-standard units. We sought to design, implement and test a methodology for the harmonization of units to the UCUM standards across a large research network.</div></div><div><h3>Methods</h3><div>Using dimensional analysis and a curated equivalence table, the proposed methodology harmonizes disparate units to UCUM standards. The process focused on identifying and converting non-UCUM conforming units, with the goal of enhancing data comparability and interoperability across different systems.</div></div><div><h3>Results</h3><div>The methodology successfully achieved over 90% coverage of laboratory data with units in UCUM standards across the TriNetX research network, a significant improvement from baseline measurements. This enhancement in unit standardization directly contributed to increased interoperability of laboratory data, facilitating more reliable and comparable data analysis across various healthcare organizations.</div></div><div><h3>Conclusion</h3><div>The successful harmonization of laboratory data units to UCUM standards represents a significant advancement in the field of biomedical informatics. By demonstrating a practical and effective approach to overcoming the challenges of non-standardized units, our study contributes to the broader efforts to improve data interoperability and usability for secondary purposes such as research and observational studies. Future work will focus on addressing the remaining gaps in unit standardization and exploring the implications of this methodology on clinical outcomes and research capabilities.</div></div>","PeriodicalId":54950,"journal":{"name":"International Journal of Medical Informatics","volume":null,"pages":null},"PeriodicalIF":3.7,"publicationDate":"2024-10-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142579093","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0