JAMIA Open最新文献

Importance: Data governance, the policies, and procedures for managing data, is a critical factor for secondary use of clinical data for research.

Objectives: This paper describes the evolution of an academic health-care organization's data governance for research, development of an external data sharing process, implementation of related processes, continuous improvement, and ongoing observations of data governance maturity.

Materials and methods: The program was designed to improve the access to and sharing of real-world data for research. Using a combination of qualitative and quantitative methods, we evaluated the program's effectiveness.

Results: Our results describe a significant improvement in data accessibility as seen in new data-driven performance indicators and in data understanding indicated by new processes, policies, and strategies.

Discussion: The paper outlines the development of a data governance process at an academic health center to support external data sharing, emphasizing the importance of data literacy, cross-office collaboration, and structured workflows to manage complex review requirements. The formalized process improved data access, identified gaps, and enabled continuous quality improvement, though it introduced new bottlenecks and required navigating multi-office reviews and researcher education.

Conclusion: These findings suggest data governance practices that may apply to other institutions.

Objective: Falls are a leading cause of morbidity and mortality among older adults. Common methods for identifying fall-related ED visits within both claims and electronic health record datasets rely on diagnosis code-based definitions, which underestimate the true prevalence of falls. This study applies a natural language processing (NLP) algorithm to ED provider notes to identify patients presenting due to falls and compares the characteristics of NLP-identified cases to those identified through diagnosis codes to identify the impact of identification strategy.

Materials and methods: This cross-sectional study analyzed ED encounter data from older adult patients who visited an ED between December 2016 and 2020. The NLP algorithm identified falls based on provider notes, searching for keywords related to falls and excluding negated and spurious matches. We also applied common ICD code methods to identify falls.

Results: We processed 50 153 ED encounters and the NLP approach identified 14 604 encounters for patients who fell. Of those, 7086 (49%) were not identified using external cause of morbidity ICD codes. Patients identified by just the NLP algorithm exhibited higher Elixhauser comorbidity scores and increased likelihood of 30-day mortality. Patients identified by NLP algorithm but not ICD codes were more likely to have severe underlying conditions such as sepsis or acute kidney disease rather than traumatic injuries.

Discussion: The NLP algorithm identifies many fall-related visits not identified by traditional methods.

Conclusion: If the causal relationships between falls and comorbid conditions are not considered in NLP algorithms, they can easily identify patients who fell, but the fall was a sequela of underlying medical illness.

Objectives: To develop and apply a reproducible methodology for evaluating generative artificial intelligence (AI) powered systems in health care, addressing the gap between theoretical evaluation frameworks and practical implementation guidance.

Materials and methods: A 5-dimension evaluation framework was developed to assess query comprehension and response helpfulness, correctness, completeness, and potential clinical harm. The framework was applied to evaluate ClinicalKey AI using queries drawn from user logs, a benchmark dataset, and subject matter expert curated queries. Forty-one board-certified physicians and pharmacists were recruited to independently evaluate query-response pairs. An agreement protocol using the mode and modified Delphi method resolved disagreements in evaluation scores.

Results: Of 633 queries, 614 (96.99%) produced evaluable responses, with subject matter experts completing evaluations of 426 query-response pairs. Results demonstrated high rates of response correctness (95.5%) and query comprehension (98.6%), with 94.4% of responses rated as helpful. Two responses (0.47%) received scores indicating potential clinical harm. Pairwise consensus occurred in 60.6% of evaluations, with remaining cases requiring third tie-breaker review.

Discussion: The framework demonstrated effectiveness in quantifying performance through comprehensive evaluation dimensions and structured scoring resolution methods. Key strengths included representative query sampling, standardized rating scales, and robust subject matter expert agreement protocols. Challenges emerged in managing subjective assessments of open-ended responses and achieving consensus on potential harm classification.

Conclusion: This framework offers a reproducible methodology for evaluating health-care generative AI systems, establishing foundational processes that can inform future efforts while supporting the implementation of generative AI applications in clinical settings.

Objective: Does a Tree-of-Thought prompt and reconsideration of Isabel Pro's differential improve ChatGPT-4's accuracy; does increasing expert panel size improve ChatGPT-4's accuracy; does ChatGPT-4 produce consistent outputs in sequential requests; what is the frequency of fabricated references?

Materials and methods: Isabel Pro, a computerized diagnostic decision support system, and ChatGPT-4, a large language model. Using 201 cases from the New England Journal of Medicine, each system produced a differential diagnosis ranked by likelihood. Statistics were Mean Reciprocal Rank, Recall at Rank, Average Rank, Number of Correct Diagnoses, and Rank Improvement. For reproducibility, the study compared the initial expert panel run to each subsequent run, using the r-squared calculation from a scatter plot of each run.

Results: ChatGPT-4 improved MRR and Recall at 10 to 0.72 but produced fewer correct diagnoses and lower average rank. Reconsideration of the Isabel Pro differential produced an improvement in Recall at 10 of 11%. The expert panel size of two produced the best result. The reproducibility runs were within 4% on average for Recall at 10, but the scatterplots showed an r-squared ranging from 0.44 to 034, suggesting poor reproducibility. Reference accuracy was 34.8% for citations and 37.8% for DOIs.

Discussion: ChatGPT-4 performs well with images and electrocardiography and in administrative practice management, but diagnosis has not proven as promising.

Conclusions: As noted above, the results demonstrate concerns for diagnostic accuracy, reproducibility, and reference citation accuracy. Until these issues are resolved, clinical usage for diagnosis will be minimal, if at all.

Objective: To describe challenges and solutions for calculating longitudinal daily opioid dose in morphine milligram equivalents from electronic health record prescriptions for a clinical trial of voluntary opioid reduction in patients with chronic non-cancer pain.

Materials and methods: Researchers obtained opioid prescriptions for 525 participants from the National Patient-Centered Clinical Research Network datamart at three health systems. Daily opioid dose was calculated using dose conversions and summing across prescriptions after applying assumptions, reviewing suspect prescribing patterns, and removing spurious prescriptions.

Results: Out of 16 071 extracted prescriptions, 1207 (8%) were unusable, and 14 864 (92%) were analyzed.

Discussion: Numerous challenges were identified related to incomplete data, inaccurate refill dates, and overlapping or duplicate prescriptions.

Conclusion: Using electronic prescription data to calculate daily doses of opioid consumption is challenging and requires significant cleaning prior to use in research. This paper recommends steps to review and clean electronic opioid prescription data.

Objectives: To develop CigStopper, a real-time, automated medical billing prototype designed to identify eligible tobacco cessation care codes, thereby reducing administrative workload while improving billing accuracy.

Materials and methods: ChatGPT prompt engineering generated a synthetic corpus of physician-style clinical notes categorized for CPT codes 99406/99407. Practicing clinicians annotated the dataset to train multiple machine learning (ML) models focused on accurately predicting billing code eligibility.

Results: Decision tree and random forest models performed best. Mean performance across all models: PRC AUC = 0.857, F1 score = 0.835. Generalizability testing on deidentified notes confirmed that tree-based models performed best.

Discussion: CigStopper shows promise for streamlining manual billing inefficiencies that hinder tobacco cessation care. ML methods lay the groundwork for clinical implementation based on good performance using synthetic data. Automating high-volume, low-value tasks simplify complexities in a multi-payer system and promote financial sustainability for healthcare practices.

Conclusion: CigStopper validates foundational methods for automating the discernment of appropriate billing codes for eligible smoking cessation counseling care.

Objectives: This study aimed to systematically evaluate and compare the diagnostic performance of leading large language models (LLMs) in common and complex clinical scenarios, assessing their potential for enhancing clinical reasoning and diagnostic accuracy in authentic clinical decision-making processes.

Materials and methods: Diagnostic capabilities of advanced LLMs (Anthropic's Claude, OpenAI's GPT variants, Google's Gemini) were assessed using 60 common cases and 104 complex, real-world cases from Clinical Problem Solvers' morning rounds. Clinical details were disclosed in stages, mirroring authentic clinical decision-making. Models were evaluated on primary and differential diagnosis accuracy at each stage.

Results: Advanced LLMs showed high diagnostic accuracy (>90%) in common scenarios, with Claude 3.7 achieving perfect accuracy (100%) in certain conditions. In complex cases, Claude 3.7 achieved the highest accuracy (83.3%) at the final diagnostic stage, significantly outperforming smaller models. Smaller models notably performed well in common scenarios, matching the performance of larger models.

Discussion: This study evaluated leading LLMs for diagnostic accuracy using staged information disclosure, mirroring real-world practice. Notably, Claude 3.7 Sonnet was the top performer. Employing a novel LLM-based evaluation method for large-scale analysis, the research highlights artificial intelligence's (AI's) potential to enhance diagnostics. It underscores the need for useful frameworks to translate accuracy into clinical impact and integrate AI into medical education.

Conclusion: Leading LLMs show remarkable diagnostic accuracy in diverse clinical cases. To fully realize their potential for improving patient care, we must now focus on creating practical implementation frameworks and translational research to integrate these powerful AI tools into medicine.

Objective: We used clinical decision support (CDS) to promote compliance with the 21st Century Cures Act's mandate that, with few exceptions, patients be granted timely access to their clinical notes.

Materials and methods: We conducted an observational analysis of note sharing rates in a large regional health system from February 2, 2021 to October 3, 2023. Throughout the study period, notes were shared with patients by default with the option not to grant note access; starting week 10, clinicians not sharing notes were presented with "hard-stop" CDS requiring selection of an allowable exception reason. Trends were examined with forward step-segmented linear regression.

Results: 0.7% of all notes were unshared; rates of unshared notes were highest in pediatrics (4.9%) and psychiatry (2.2%). Rates dropped substantially following hard-stop CDS introduction (downward step of 0.96%; 95% CI -1.17 to -0.024). Despite high portal access (72.6%), few notes were viewed by patients/proxies (17.0%).

Discussion: We found very low overall rates of unshared notes; the significant drop in the rates of unshared notes following the introduction of hard-stop CDS is consistent with prior research showing that hard-stop CDS can be an effective tool. The higher rates of unshared notes in pediatrics and psychiatry likely reflect considerations around sensitive information that are inherent to these fields.

Conclusions: CDS effectively promoted note sharing, but patient engagement remained low.

Objectives: We analyzed interoperability-related Real World Testing results to identify whether developers are providing meaningful results with the appropriate context to enable stakeholders to understand the Certified Health IT conformance and interoperability when deployed in production environments.

Materials and methods: This qualitative study analyzed components of the Assistant Secretary for Technology Policy's transitions of care criterion Real World Testing results of 5 inpatient and 5 ambulatory health IT developers with the largest market share.

Results: Developers provided interoperability measures; however, none of the developers' presented results in a meaningful way with the appropriate context to understand product interoperability.

Discussion: Our results suggest that developers with ASTP/Office of the National Coordinator (ONC) Certified Health IT modules are not providing interoperability transparency through Real World Testing as required by the ONC Health IT Certification Program and intended by the 21st Century Cures Act.

Conclusion: Clearer developer guidance and actual metric requirements on Real World Testing may be required and the authorized certification bodies, who review developer results, may need to more closely inspect reports to look at the quality of reported results.

Objectives: In response to requests from several Rapid Acceleration of Diagnostics-Underserved Populations (RADx-UP) community-engaged research projects to include Long COVID common data elements (CDEs) in the existing RADx-UP CDEs, the RADx-UP Coordination and Data Collection Center (CDCC) leadership formed the Long COVID CDEs Task Force.

Materials and methods: The Task Force, composed mainly of community partners and RADx-UP project members, participated in various activities to evaluate the Long COVID CDEs fit for purpose from the Researching COVID to Enhance Recovery (RECOVER) program for RADx-UP use.

Results and discussion: The Task Force's efforts led to a compilation of lessons learned and the creation of a novel set of 28 CDEs that are appropriate for community-engaged research in Long COVID.

Conclusion: Utilization of standardized CDEs does not always work for the communities involved in the research, but creation of a community-involved task force can lead to a meaningful, rich set of CDEs.