JAMIA Open最新文献

Objective: Does a Tree-of-Thought prompt and reconsideration of Isabel Pro's differential improve ChatGPT-4's accuracy; does increasing expert panel size improve ChatGPT-4's accuracy; does ChatGPT-4 produce consistent outputs in sequential requests; what is the frequency of fabricated references?

Materials and methods: Isabel Pro, a computerized diagnostic decision support system, and ChatGPT-4, a large language model. Using 201 cases from the New England Journal of Medicine, each system produced a differential diagnosis ranked by likelihood. Statistics were Mean Reciprocal Rank, Recall at Rank, Average Rank, Number of Correct Diagnoses, and Rank Improvement. For reproducibility, the study compared the initial expert panel run to each subsequent run, using the r-squared calculation from a scatter plot of each run.

Results: ChatGPT-4 improved MRR and Recall at 10 to 0.72 but produced fewer correct diagnoses and lower average rank. Reconsideration of the Isabel Pro differential produced an improvement in Recall at 10 of 11%. The expert panel size of two produced the best result. The reproducibility runs were within 4% on average for Recall at 10, but the scatterplots showed an r-squared ranging from 0.44 to 034, suggesting poor reproducibility. Reference accuracy was 34.8% for citations and 37.8% for DOIs.

Discussion: ChatGPT-4 performs well with images and electrocardiography and in administrative practice management, but diagnosis has not proven as promising.

Conclusions: As noted above, the results demonstrate concerns for diagnostic accuracy, reproducibility, and reference citation accuracy. Until these issues are resolved, clinical usage for diagnosis will be minimal, if at all.

Objective: To describe challenges and solutions for calculating longitudinal daily opioid dose in morphine milligram equivalents from electronic health record prescriptions for a clinical trial of voluntary opioid reduction in patients with chronic non-cancer pain.

Materials and methods: Researchers obtained opioid prescriptions for 525 participants from the National Patient-Centered Clinical Research Network datamart at three health systems. Daily opioid dose was calculated using dose conversions and summing across prescriptions after applying assumptions, reviewing suspect prescribing patterns, and removing spurious prescriptions.

Results: Out of 16 071 extracted prescriptions, 1207 (8%) were unusable, and 14 864 (92%) were analyzed.

Discussion: Numerous challenges were identified related to incomplete data, inaccurate refill dates, and overlapping or duplicate prescriptions.

Conclusion: Using electronic prescription data to calculate daily doses of opioid consumption is challenging and requires significant cleaning prior to use in research. This paper recommends steps to review and clean electronic opioid prescription data.

Objectives: To develop CigStopper, a real-time, automated medical billing prototype designed to identify eligible tobacco cessation care codes, thereby reducing administrative workload while improving billing accuracy.

Materials and methods: ChatGPT prompt engineering generated a synthetic corpus of physician-style clinical notes categorized for CPT codes 99406/99407. Practicing clinicians annotated the dataset to train multiple machine learning (ML) models focused on accurately predicting billing code eligibility.

Results: Decision tree and random forest models performed best. Mean performance across all models: PRC AUC = 0.857, F1 score = 0.835. Generalizability testing on deidentified notes confirmed that tree-based models performed best.

Discussion: CigStopper shows promise for streamlining manual billing inefficiencies that hinder tobacco cessation care. ML methods lay the groundwork for clinical implementation based on good performance using synthetic data. Automating high-volume, low-value tasks simplify complexities in a multi-payer system and promote financial sustainability for healthcare practices.

Conclusion: CigStopper validates foundational methods for automating the discernment of appropriate billing codes for eligible smoking cessation counseling care.

Objectives: This study aimed to systematically evaluate and compare the diagnostic performance of leading large language models (LLMs) in common and complex clinical scenarios, assessing their potential for enhancing clinical reasoning and diagnostic accuracy in authentic clinical decision-making processes.

Materials and methods: Diagnostic capabilities of advanced LLMs (Anthropic's Claude, OpenAI's GPT variants, Google's Gemini) were assessed using 60 common cases and 104 complex, real-world cases from Clinical Problem Solvers' morning rounds. Clinical details were disclosed in stages, mirroring authentic clinical decision-making. Models were evaluated on primary and differential diagnosis accuracy at each stage.

Results: Advanced LLMs showed high diagnostic accuracy (>90%) in common scenarios, with Claude 3.7 achieving perfect accuracy (100%) in certain conditions. In complex cases, Claude 3.7 achieved the highest accuracy (83.3%) at the final diagnostic stage, significantly outperforming smaller models. Smaller models notably performed well in common scenarios, matching the performance of larger models.

Discussion: This study evaluated leading LLMs for diagnostic accuracy using staged information disclosure, mirroring real-world practice. Notably, Claude 3.7 Sonnet was the top performer. Employing a novel LLM-based evaluation method for large-scale analysis, the research highlights artificial intelligence's (AI's) potential to enhance diagnostics. It underscores the need for useful frameworks to translate accuracy into clinical impact and integrate AI into medical education.

Conclusion: Leading LLMs show remarkable diagnostic accuracy in diverse clinical cases. To fully realize their potential for improving patient care, we must now focus on creating practical implementation frameworks and translational research to integrate these powerful AI tools into medicine.

Objective: We used clinical decision support (CDS) to promote compliance with the 21st Century Cures Act's mandate that, with few exceptions, patients be granted timely access to their clinical notes.

Materials and methods: We conducted an observational analysis of note sharing rates in a large regional health system from February 2, 2021 to October 3, 2023. Throughout the study period, notes were shared with patients by default with the option not to grant note access; starting week 10, clinicians not sharing notes were presented with "hard-stop" CDS requiring selection of an allowable exception reason. Trends were examined with forward step-segmented linear regression.

Results: 0.7% of all notes were unshared; rates of unshared notes were highest in pediatrics (4.9%) and psychiatry (2.2%). Rates dropped substantially following hard-stop CDS introduction (downward step of 0.96%; 95% CI -1.17 to -0.024). Despite high portal access (72.6%), few notes were viewed by patients/proxies (17.0%).

Discussion: We found very low overall rates of unshared notes; the significant drop in the rates of unshared notes following the introduction of hard-stop CDS is consistent with prior research showing that hard-stop CDS can be an effective tool. The higher rates of unshared notes in pediatrics and psychiatry likely reflect considerations around sensitive information that are inherent to these fields.

Conclusions: CDS effectively promoted note sharing, but patient engagement remained low.

Objectives: We analyzed interoperability-related Real World Testing results to identify whether developers are providing meaningful results with the appropriate context to enable stakeholders to understand the Certified Health IT conformance and interoperability when deployed in production environments.

Materials and methods: This qualitative study analyzed components of the Assistant Secretary for Technology Policy's transitions of care criterion Real World Testing results of 5 inpatient and 5 ambulatory health IT developers with the largest market share.

Results: Developers provided interoperability measures; however, none of the developers' presented results in a meaningful way with the appropriate context to understand product interoperability.

Discussion: Our results suggest that developers with ASTP/Office of the National Coordinator (ONC) Certified Health IT modules are not providing interoperability transparency through Real World Testing as required by the ONC Health IT Certification Program and intended by the 21st Century Cures Act.

Conclusion: Clearer developer guidance and actual metric requirements on Real World Testing may be required and the authorized certification bodies, who review developer results, may need to more closely inspect reports to look at the quality of reported results.

Objectives: In response to requests from several Rapid Acceleration of Diagnostics-Underserved Populations (RADx-UP) community-engaged research projects to include Long COVID common data elements (CDEs) in the existing RADx-UP CDEs, the RADx-UP Coordination and Data Collection Center (CDCC) leadership formed the Long COVID CDEs Task Force.

Materials and methods: The Task Force, composed mainly of community partners and RADx-UP project members, participated in various activities to evaluate the Long COVID CDEs fit for purpose from the Researching COVID to Enhance Recovery (RECOVER) program for RADx-UP use.

Results and discussion: The Task Force's efforts led to a compilation of lessons learned and the creation of a novel set of 28 CDEs that are appropriate for community-engaged research in Long COVID.

Conclusion: Utilization of standardized CDEs does not always work for the communities involved in the research, but creation of a community-involved task force can lead to a meaningful, rich set of CDEs.

Objective: Little is known about how clinical decision support (CDS) tools can support care teams in changing clinical decisions to account for patients' social risks. We piloted a suite of electronic health record (EHR)-based CDS tools designed to facilitate social risk-informed care decisions to assess how the tools were used in practice and how they could be improved.

Materials and methods: After developing CDS tools through a process involving clinic staff and patient engagement, the tools were implemented in three community health center clinics. Data from staff interviews, observations of meetings with clinic staff, and the EHR were used to understand tool use patterns, and to yield insights that were then used to inform tool revisions.

Results: The overarching suggestion derived from the study data was that the tools should shift from making care recommendations to instead supporting documentation of social risk-related actions that clinical team members had already taken. Other revisions were guided by four additional insights: the CDS tools should: (1) facilitate documentation in standardized, short formats, (2) make documentation easy and consistent, (3) support work distribution across care team members, and (4) ensure documentation could serve multiple purposes.

Discussion: The CDS tools were revised to improve usefulness and acceptability for primary care teams in community clinics that serve patients with social risks. Numerous challenges exist in designing tools that can accommodate diverse clinics and workflows.

Conclusion: These findings provide insights on how CDS tools can be optimized for social risk-informed care while minimizing care team burdens.

Objectives: This study aims to explore multimorbidity patterns associated with diabetes by analyzing user engagement in online diabetes support communities and their interactions with other disease-related communities. Additionally, it seeks to assess whether early signals of diabetes can be detected through online engagement data.

Materials and methods: We collected Reddit data for 3 primary diabetes-related subreddits ("diabetes," "diabetes_t1," and "diabetes_t2") and 88 other disease-related subreddits from 2008 to 2024. A bipartite network was constructed linking users to subreddits, which was then transformed into a weighted multimorbidity network. Significant links were identified using a statistical threshold to ensure meaningful connections between subreddits. Additionally, we analyzed user engagement timelines to identify potential early signals of diabetes.

Results: Diabetes is strongly linked to mental health conditions (such as depression, anxiety, and ADHD) and weight management discussions. Other notable associations include autoimmune diseases, chronic pain, gastrointestinal disorders, and reproductive health issues. Early signals of type 2 diabetes were detected in mental health, obesity, and pregnancy conditions, but no significant early indicators were found for type 1 diabetes.

Discussion: This study is the first large-scale empirical analysis of multimorbidity patterns and early signals of diabetes in online communities. The findings reinforce the known multimorbidity of diabetes, particularly its ties to mental health and obesity. The presence of early signals suggests that social media data could help identify individuals at risk before diagnosis, offering opportunities for early intervention.

Conclusion: Our findings demonstrate that social media data can reveal both multimorbidity patterns and early signals of diabetes, offering insights beyond traditional health records. As digital health data continue to grow, effectively leveraging these resources will become increasingly important for advancing diabetes prevention and management.

Objective: Usage of autotext or "dotphrases" is ubiquitous among provider workflows in electronic health records (EHRs). Yet, little is known about the impact of these tools in inpatient settings and among resident physicians. We aimed to evaluate the association between autotext usage and documentation time among resident physicians in an academic medical center using the Cerner EHR.

Materials and methods: The association between autotext executions and documentation time per patient seen for 705 resident physicians rotating at a large academic medical center from July 2021 to June 2023 was analyzed via linear regression after controlling for specialty, post-graduate year (PGY), provider gender and patient volume.

Results: There was no significant overall association between autotext executions per patient seen and documentation time per patient seen in specialties using Dynamic Documentation as their primary workflow (β=-0.1 min per autotext execution per patient seen, 95% CI -0.6 to 0.5 min, P =.79). However, there was increased documentation time among residents with no autotext usage compared to residents who used autotext, and this effect was mediated by use of personalized autotexts. Specialty, PGY, gender and patient volume were significant determinants of documentation time.

Discussion: Efforts to decrease documentation time among resident physicians should encourage autotext adoption but should not be focused on promotion of autotext usage alone. Further research should address the questions of identifying other determinants of documentation time, autotext design standards, and how autotext usage affects measures of note quality.

Conclusion: Autotext adoption decreases documentation time among resident physicians, but among those who adopt autotext, higher levels of usage show no benefit.