JAMIA Open最新文献

Objectives: To assess the interpretability and acceptance of Shapley values for making artificial intelligence/machine learning (AI/ML) tools more transparent, interpretable, and useful to clinicians.

Materials and methods: Structured assessments were conducted with 30 clinicians (15 providers; 15 nurses; 8 assessments per clinician) to evaluate their ability to understand interventional Shapley Additive exPlanations (SHAP) values, a type of Shapley value that provides individualized variable importance scores and ascertain their perspective on SHAP value utility for the use of an AI/ML sepsis diagnostic. Participants were shown the diagnostic interface for real clinical scenarios with de-identified patient data with and without SHAP values. The primary outcomes were clinician ability to correctly interpret SHAP values and clinician self-reported improvement in their understanding of how the AI/ML algorithm produced its result.

Results: Participants correctly interpreted SHAP values in 235 of 240 assessments (98%; CI, 95%-99%) and reported SHAP values improved their understanding of how the algorithm produced its result in every case (240/240; 100%; CI, 99%-100%). Participants were unanimous (30/30) in preferring the interface with SHAP values over the interface without.

Discussion: Clinician participants strongly preferred the device interface with SHAP values, were unanimous in reporting SHAP values improved their understanding of the AI/ML diagnostic, and scored nearly perfectly when asked to interpret SHAP values.

Conclusion: These results suggest health care providers value transparency into AI/ML algorithms designed for clinical use, and that Shapley values are a useful approach to providing that transparency, which in turn may improve tool adoption and clinical utility.

Objective: To explore the relationship between ambulatory physician electronic health record (EHR) use characteristics and proxies for physician efficiency.

Materials and methods: A longitudinal cohort study was conducted to examine physician-month EHR use metadata in 413 US organizations between May 2019 and April 2022. A multi-model machine learning classifier was developed to predict physician efficiency. The main outcomes of the study were physician efficiency, measured as the proportion of same-day chart completion by specialty, and productivity, measured as daily patient visit volume, both segmented into quintiles.

Results: The study included 218 610 unique physicians with 5 193 385 physician-month observations from 413 organizations with an average chart completion efficiency of 72.9% and 10.8 visits per scheduled day. The primary ML analysis achieved an accuracy of 0.74 in classifying physician-months with high chart completion efficiency and highlighted associations with key features, such as inbox message turnaround time <1.5 days and after-hours documentation <25 min/scheduled day. A secondary analysis achieved an accuracy of 0.84 in classifying physician-months with high visit volumes, indicating that factors such as EHR time outside scheduled hours <4.1 min/visit and clinical review time <3.2 min/visit were associated with higher visit volumes.

Discussion and conclusion: Implementing specific EHR use measures with distinct thresholds, such as inbox management and after-hours documentation, could help target interventions to enhance productivity, providing actionable insights to create balanced and efficient work environments that improve patient care and reduce EHR time.

[This corrects the article DOI: 10.1093/jamiaopen/ooaf134.].

Objective: This study evaluates the usefulness of explicit syntactic knowledge, integrated via a neural mechanism, in improving the accuracy of named entity recognition in the domain of biomedical text processing.

Materials and methods: Syntactic structure of a text can be helpful to determine whether a certain part of the text is an entity or not. Parsing is an essential technique in natural language processing (NLP) that can be utilized to determine the syntactic structure of sentences in human languages. We propose to infuse syntactic knowledge through the attention mechanism using dependency parsing and sequence labelling parsing, as well as the multi-task learning paradigm. Experiments were conducted on five datasets: MTSamples, VAERS, NCBI-disease, BC2GM, and JNLPBA.

Results: We demonstrate improvements in the F1 score over the current state of the art on 3 out of 5 datasets (MTSamples, VAERS, and NCBI).

Discussion: We reduce the number of mismatches with gold labels in particular in the n-dash and parentheses tokens and in compound and adjective modifier dependencies.

Conclusion: Syntactic features improve NER accuracy in attention-based neural systems, and parsing as sequence labelling brings additional benefits.

Objectives: Electronic health record (EHR) phenotyping often relies on noisy proxy labels, which undermine the reliability of downstream risk prediction. Active learning can reduce annotation costs, but typical heuristics do not directly optimize downstream prediction. Our goal was to develop a framework that directly uses downstream prediction performance as feedback to guide phenotype correction and sample selection under constrained labeling budgets.

Materials and methods: We propose reinforcement-enhanced label-efficient active phenotyping (RELEAP), a reinforcement learning-based active learning framework. Reinforcement-enhanced label-efficient adaptively integrates multiple querying strategies and, unlike prior methods, updates its policy based on feedback from downstream models. We evaluated RELEAP on a de-identified Duke University Health System (DUHS) cohort (2014-2024) for incident lung cancer risk prediction, using logistic regression and penalized Cox survival models. Performance was benchmarked against noisy-label baselines and single-strategy active learning.

Results: Reinforcement-enhanced label-efficient improved over the proxy-only baseline and approached oracle performance under the same budget. Logistic AUC increased from 0.774 to 0.807. Survival concordance index increased from 0.715 to 0.749. Gains were stable across iterations using downstream feedback. These trends were consistent in sex-stratified subgroup analyses (female vs male).

Discussion: By linking phenotype refinement to prediction outcomes, RELEAP learns which samples most improve downstream discrimination and calibration, offering a more principled alternative to fixed active learning rules.

Conclusion: Reinforcement-enhanced label-efficient optimizes phenotype correction through downstream feedback, offering a scalable, label-efficient paradigm that reduces manual chart review and enhances the reliability of EHR-based risk prediction.

Objectives: To develop and validate an agent-based Large Language Model (LLM) system for extracting structured data from breast cancer synoptic pathology reports and assess the performance gap between synthetic and real-world validation.

Materials and methods: We developed a modular artificial intelligence (AI) agent-based framework employing sequential specialized LLMs for parsing pathology reports and extracting structured data. We normalized College of American Pathologists (CAP) cancer protocols into 8 sections, 86 subsections, and 229 discrete fields. Seven leading LLMs (gemini-2.5-pro, llama3.3-70b, phi4-14b, deepseek-r1 14B/70B, gemma3-27b, gemini-2.0-flash-lite) were validated using dual evaluation: synthetic validation (864 controlled test cases) and real-world ground truth (6651 annotated fields from 90 pathology reports).

Results: Synthetic validation demonstrated strong performance (accuracy: 93.8%-99.0%). Real-world evaluation revealed field extraction recall ranging from 61.8% to 87.7%, demonstrating a substantial "reality gap" with performance drops of 11-32 percentage points. The gemini-2.5-pro model achieved the highest real-world recall (87.7%). Model size did not predict performance: the 14B-parameter deepseek-r1 (77.6%) outperformed its 70B-parameter counterpart (70.4%).

Discussion: The substantial performance degradation from synthetic to real-world data underscores the complexity of authentic clinical documentation. Smaller models can achieve competitive or superior recall, reducing computational costs. With even the best models missing 12%-38% of annotated fields, mandatory human verification is essential for clinical deployment.

Conclusion: While LLM-based extraction systems show promise for pathology data extraction, synthetic validation alone provides false confidence. Rigorous real-world ground truth evaluation with expert annotation is essential before clinical deployment. These systems are best positioned as screening tools with mandatory human oversight rather than autonomous decision-making systems.

Background: Traditional clinical trial enrollment relies on manual screening and coordinator-led recruitment, creating scalability barriers in high-volume perioperative environments. This study evaluated whether a fully automated, electronic health record (EHR)-integrated clinical decision support (CDS) system could identify eligible patients and engage clinicians in real time without manual screening or dedicated research staff.

Methods: In this prospective implementation study, predefined respiratory-risk criteria were computed within the UCLA Perioperative Data Warehouse and transmitted to the EHR via Healthcare Level Seven interfaces. Patients meeting inclusion criteria automatically triggered Best Practice Advisories (BPAs) recommending an intervention. Outcomes included system accuracy in eligibility identification, provider adherence to BPA recommendations, and technical performance metrics.

Results: The automated system processed 10 592 eligible patients and achieved 51.2% provider adherence (5424 patients) to CDS prompts without coordinator involvement. BPA allocation accuracy was 69.7% among patients recovering in the post-anesthesia care unit and 59.4% when including unanticipated ICU transfers. Adherence varied significantly by care team composition, with full teams (attending + CRNA + resident) achieving 57.4% adherence compared with 42.2% for solo attendings. Workflow factors were stronger predictors of adherence than patient clinical characteristics, indicating minimal selection bias.

Conclusions: Fully automated, EHR-integrated CDS can enable large-scale, workflow-embedded enrollment into implementation-focused studies. While not a substitute for research designs requiring consent or randomization, this framework demonstrates a scalable approach for automated prescreening and CDS-driven prompting that reduces reliance on coordinator-dependent processes and supports real-world implementation science.

Objectives: Electronic health record (EHR) order preference lists and order sets potentially improve efficiency but have limited utility in complex primary care settings. We assessed adoption, impact on ordering efficiency, and clinician perceptions of a comprehensive set of nested order panels (xOrders) for adult primary care.

Methods: In Phase 1 (gradual implementation), 404 xOrders were released (November 29, 2020-September 25, 2021). Beginning of Phase 2 (rapid implementation), 630 xOrders were released with an additional 253 xOrders added (September 26, 2021-June 24, 2023). Three outcomes captured adoption: xOrders used per week; number of clinician users per week; and percent of xOrders of all orders. Impact of xOrders on times in orders per encounter per clinician was evaluated with mixed effects interrupted time series. t-Tests evaluated differences between low, moderate, and high utilizers. A survey captured clinicians' perceptions in November 2022.

Results: xOrders were used 536 (SD, 245) times/week and by 57(15) clinicians/week in Phase 2. xOrders as a percent of all orders ranged from 0% to 31% across clinicians. Time spent in orders per encounter decreased by 14 ± 5 s (P =.01) from Phase 1 to 2 for high utilizers, decreased by 7(3) s (P=.05) for moderate utilizers, and increased by 1(3) s for low utilizers (P=.81); low and high utilizers were significantly different (P=.02). Most (77%) survey respondents agreed that xOrders improved ordering efficiency.

Discussion and conclusions: Despite yielding time savings and positive clinician feedback, the xOrder intervention showed limited adoption and impact, suggesting the need for expanded content and increased adoption to realize larger efficiency gains.

Objectives: To develop a multimodal pediatric critical care datamart supporting predictive modeling and decision support tool development, integrating high-resolution physiologic and clinical data and future clinical deployment.

Materials and methods: We developed a continuously expanding datamart integrating electronic health record data, high-resolution telemetry, and extracorporeal membrane oxygenation (ECMO)-domain datasets. The platform links static and longitudinal time-series variables with expert-curated neurologic outcomes for both ECMO and non-ECMO patients, enabling trajectory-based analyses.

Results: The datamart currently includes 25 762 pediatric patients, of whom 395 received ECMO support. The datamart captures granular longitudinal physiologic, laboratory, medication, and telemetry data suitable for dynamic predictive modeling.

Discussion: Existing ECMO prognostication tools rely on static variables and lack appropriate control cohorts. This datamart enables trajectory-based multimodal modeling, that reflects evolving physiology and neurologic outcomes.

Conclusion: This platform provides a scalable foundation for predictive modeling across pediatric critical care, beyond ECMO, to support precision decision-making and outcomes research.

Objectives: This study aims to develop a detailed understanding of provider and Information Technology (IT) operations staff experiences and attitudes regarding patients' ability to edit their data. This includes understanding barriers to developing a process to write back data into the electronic health record (EHR) as well as a concrete set of recommendations on incorporating patient-generated data into the EHR.

Materials and methods: RealRisks, our team's Fast Healthcare Interoperability Resources-compliant web-based patient decision aid, was utilized as an exemplar platform in which patients can access EHR data and review, correct, and contribute patient-derived data when specific elements are missing. An interview guide was developed and semi-structured interviews of 9 participants (physicians n = 4, IT operations staff n = 5) at Columbia University Irving Medical Center were carried out to understand the feasibility of writing back patient-entered edits into the EHR using the RealRisks decision aid.

Results: Providers and IT operations staff reported varied knowledge of how patients interact with their data but collectively stated a need for increasing EHR accuracy that prioritizes provider-patient communication. Participants supported a write-back process and had specific suggestions for implementation mechanisms (such as the option to upload test results when submitting changes).

Discussion: Providers and IT operations staff maintained that existing data management routes used for external data incorporation should be utilized, and that providers should screen edit requests to ensure EHR quality and accuracy.

Conclusion: While participants felt a write-back of patient-derived data would be helpful, future studies should directly assess nursing staff and advanced practice providers as well as patient perspectives to ensure equity and efficacy.