JAMIA Open最新文献

Objectives: To evaluate ChatGPT's ability to perform thematic analysis of medical Best Practice Advisory (BPA) free-text comments and identify prompt engineering strategies that optimize performance.

Materials and methods: We analyzed 778 BPA comments from a pilot AI-enabled clinical deterioration intervention at Stanford Hospital, categorized as reasons for deterioration (Category 1) and care team actions (Category 2). Prompt engineering strategies (role, context specification, stepwise instructions, few-shot prompting, and dialogue-based calibration) were tested on a 20% random subsample to determine the best-performing prompt. Using that prompt, ChatGPT conducted deductive coding on the full dataset followed by inductive analysis. Agreement with human coding was assessed as inter-rater reliability (IRR) using Cohen's Kappa (κ).

Results: With structured prompts and calibration, ChatGPT achieved substantial agreement with human coding (κ = 0.76 for Category 1; κ = 0.78 for Category 2). Baseline agreement was higher for Category 1 than Category 2, reflecting differences in comment type and complexity, but calibration improved both. Inductive analysis yielded 9 themes, with ChatGPT-generated themes closely aligning with human coding.

Discussion: ChatGPT can accelerate qualitative analysis, but its rigor depends heavily on prompt engineering. Key strategies included role and context specification, pulse-check calibration, and safeguard techniques, which enhanced reliability and reproducibility.

Conclusion: This study demonstrates the feasibility of ChatGPT-assisted thematic analysis and introduces a structured approach for applying LLMs to qualitative analysis of clinical free-text data, underscoring prompt engineering as a methodological lever.

Background: Health information exchanges (HIE), tools that electronically share clinical data across healthcare organizations, provide the opportunity to improve patient care. While widely available, HIE is utilized in only 2%-10% of patient encounters. Few studies have explored current barriers to use. The goal of this study was to evaluate current clinician perspectives on HIE and barriers to use at the point of care.

Methods: We conducted a population-based survey of internal medicine (IM) and emergency medicine (EM) physicians, physician assistants, and nurse practitioners at 8 health systems in the Atlanta area. Survey responses were analyzed overall and by specialty.

Results: Of 1239 clinicians who were invited to participate, 276 (22.3%) responded, with 65.6% of respondents working in inpatient IM and 32.6% in EM. 80.4% of respondents reported using HIE at least once a day, while 4.8% reported never using HIE. Most clinicians used HIE at least daily to access lab results (80.2%), clinical notes (81.9%), imaging reports (74.0%), and medication lists (71.2%). The most reported barriers to HIE utilization included unavailability of needed information (66.4%), adding time to patient care (45.5%), and ease of simply reordering tests (31.6%). HIE use and reported barriers to use were similar across IM and EM providers.

Conclusions: Of those responding to the survey, daily access of HIE was common. We identified several barriers to HIE use, which can be used to develop targeted interventions to improve utilization and patient care. Approaches to reach survey non-responders are also needed.

Objective: This study evaluates the performance and deployment feasibility of a machine learning (ML) model to identify adult-onset type 1 diabetes (T1D) initially coded as type 2 on electronic medical records (EMRs) from a health information exchange (HIE). To our knowledge, this is the first evaluation of such a model on real-world HIE data.

Materials and methods: An existing ML model, trained on national US EMR data, was tested on a regional HIE dataset, after several adjustments for compatibility. A localized model retrained on the regional dataset was compared to the national model. Discrepancies between the 2 datasets' features and cohorts were also investigated.

Results: The national model performed well on HIE data (AUROC = 0.751; precision at 5% recall [PR5] = 25.5%), and localization further improved performance (AUROC = 0.774; PR5 = 35.4%). Differences in the 2 models' top predictors reflected the discrepancies between the datasets and gaps in HIE data capture.

Discussion: The adjustments needed for testing on HIE data highlight the importance of aligning algorithm design with deployment needs. Moreover, localization increased precision, making it more appealing for patient screening, but added complexity and may impact scalability. Additionally, while HIEs offer opportunities for large-scale deployment, data inconsistencies across member organizations could undermine accuracy and providers' trust in ML-based tools.

Conclusion: Our findings offer valuable insights into the feasibility of at-scale deployment of ML models for high-risk patient identification. Although this work focuses on detecting potentially misclassified T1D, our learnings can also inform other applications.

Objectives: To implementation an automated multi-institutional pipeline that delivers breast-cancer risk integrated with polygenic risk scores, monogenic variants, family history, and clinical factors, emphasizing operational challenges and their solutions.

Materials and methods: A five-stage process was executed at ten sites. Data streams from REDCap surveys, PRS and monogenic reports, and MeTree pedigrees were normalized and forwarded through a REDCap plug-in to the CanRisk API.

Results: Integrated risk was returned to >10 000 women; 3.6% were ≥25 % lifetime risk and 0.9% carried pathogenic variants. Pipeline generated score aligns well with manual generated ones. Major barriers such as heterogeneous pedigree formats, missing data, edge-case handling, and evolving model versions were identified and resolved through mapping rules, imputations, and iterative testing.

Discussion: Cross-platform data harmonization and stakeholder alignment were decisive for success. Borderline-risk communication and model-version drift remain open issues.

Conclusion: Large-scale PRS-integrated breast-cancer risk reporting is feasible but requires robust interoperability standards and iterative governance.

Objective: To evaluate predictive diagnostic performance of open- and closed-source large language models (LLMs) in emergency medicine, addressing the urgent need for innovative clinical decision support tools amid rising patient volumes and staffing shortages.

Materials and methods: We generated 2370 AI-driven diagnostic predictions (Top-5 diagnoses from each of 6 model pipelines per patient), using data from 79 real-world emergency department cases collected consecutively during a 24-hour peak influx period at a tertiary care center. Pipelines combined open- and closed-source embedding models (text-embedding-ada-002, MXBAI) with foundational models (GPT-4, Llama3, and Qwen2) grounded via retrieval-augmented generation using emergency medicine textbooks. Models' predictions were assessed against reference diagnoses established by expert consensus.

Results: All pipelines achieved comparable diagnostic match rates (62.03%-72.15%). Diagnostic performance was significantly influenced by case characteristics: match rates were notably higher for specific versus unspecific diagnoses (85.53% vs 31.41%, P < .001) and surgical versus medical cases (79.49% vs 56.25%, P < .001). Open-source models demonstrated markedly superior sourcing capabilities compared to GPT-4-based combinations (P < 1.4e-12), with MBXAI/Qwen2 pipeline achieving perfect citation verification.

Discussion: Diagnostic accuracy primarily depended on case characteristics rather than the choice of model pipeline, highlighting fundamental AI alignment challenges in clinical reasoning. Low performance in unspecific diagnoses underscores inherent complexities in clinical definitions rather than technological shortcomings alone.

Conclusion: Open-source LLM pipelines provide enhanced sourcing capabilities, crucial for transparent clinical decision-making and interpretability. Further research should expand knowledge bases to include hospital guidelines and regional epidemiology, while exploring on-premises solutions to better align with privacy regulations and clinical integration.

Objective: The Bridge2AI program is establishing rules of practice for creating ethically sourced health data repositories to support the effective use of ML/AI in biomedical and behavioral research. Given the initially undefined nature of ethically sourced data, this work concurrently developed definitions and guidelines alongside repository creation, grounded in a practical, operational framework.

Materials and methods: A Value Sensitive Design (VSD) approach was used to explore ethical tensions across stages of health data repository development. The conceptual investigation drew from supply chain management (SCM) processes to (1) identify actors who would interact with or be affected by the data repository use and outcomes; (2) determine what values to consider (ie, traceability accountability, security); and (3) analyze and document value trade-offs (ie, balancing risks of harm to improvements in healthcare). This SCM framework provides operational guidance for managing complex, multi-source data flows with embedded bias mitigation strategies.

Results: This conceptual investigation identified the actors, values, and tensions that influence ethical sourcing when creating a health data repository. The SCM steps provide a scaffolding to support ethical sourcing across the pre-model stages of health data repository development. Ethical sourcing includes documenting data provenance, articulating expectations for experts, and practices for ensuring data privacy, equity, and public benefit. Challenges include risks of ethics washing and highlight the need for transparent, value-driven practices.

Discussion: Integrating VSD with SCM frameworks enables operationalization of ethical values, improving data integrity, mitigating biases, and enhancing trust. This approach highlights how foundational decisions influence repository quality and AI/ML system usability, addressing provenance, traceability, redundancy, and risk management central to ethical data sourcing.

Conclusion: To create authentic, impactful health data repositories that serve public health goals, organizations must prioritize transparency, accountability, and operational frameworks like SCM that comprehensively address the complexities and risks inherent in data stewardship.

Objective: To develop and evaluate a patient-centric medication module within a personal health record (PHR) app for capturing medication use, focusing on accuracy, usability, and concordance.

Materials and methods: The medication module offered 4 entry methods: picklist, National Drug Code (NDC), free-text, and portal import, with the first 2 leveraging RxNorm and openFDA APIs. Patients from an integrated delivery network (IDN) created medication lists and recorded daily use in the app's diary. Pharmacists evaluated medication accuracy by reviewing patient-uploaded medication images. Usability was measured using the System Usability Scale (SUS). Concordance was assessed by comparing Electronic Health Records (EHR) with diary entries.

Results: Over a 14-day period, 85 patients entered 617 medications, with 533 logged in the diary representing current use. Picklist was the most used entry method. Overall medication entry accuracy was 92% (picklist 97%; NDC 87%; free-text 84%; and portal import 100%). The mean system usability score was 56.5 for the study app (patients) and 80.8 for the medication module (pharmacists). EHR concordance with diary entries was low (25% using the 14-day window; 53% using a 1-year window); most unmatched entries were over-the-counter (OTC) medications.

Discussion: Accurate and complete medication records are essential for the safe and effective use of medications. This patient-centric medication module supported accurate capture of prescription and OTC medications. Gaps in EHR data highlight the need to improve medication record accuracy and reconciliation.

Conclusion: Patient-generated health data can have a central role in creating the "Best Possible Medication History" envisioned by the World Health Organization.

Objective: Severe maternal morbidity and mortality are higher in the United States than in other high-income countries, and unacceptable disparities persist. To facilitate research on these outcomes, we developed a standardized approach for extracting perinatal data from electronic health records (EHRs).

Materials and methods: To support data model building and validation, we harmonized perinatal EHR data in a common data model, building on lessons learned from multiple prior projects.

Results: We developed an Entity Relationship Diagram (ERD) that aggregates perinatal EHR data at appropriate granularities (ie, mothers, infants, encounters) with indexing of observations to gestational age and time from delivery. We then developed a standard approach to extract, transform, and load pregnancy-related observations from EHRs for inclusion in PCORnet^® Common Data Model tables.

Discussion: Our ERD can facilitate cross-institutional research to identify populations at risk and prompt interventions to improve perinatal outcomes.

Conclusion: A structured approach can accelerate the use of EHR data for perinatal research.

Objectives: To develop ECG-FM, an open-weight foundation model for electrocardiogram (ECG) analysis, rigorously evaluate its performance on clinically salient tasks, and openly release it alongside a public benchmark.

Materials and methods: In a study using 1.5 million 12-lead ECGs, we present ECG-FM, a transformer-based foundation model pretrained with hybrid self-supervision that combines masked reconstruction and contrastive learning with ECG-specific augmentation. Downstream, we evaluate multi-label ECG interpretation and prediction of reduced left ventricular ejection fraction (LVEF), introducing an openly available benchmark on the MIMIC-IV-ECG dataset. We assess ECG-FM's capabilities through data scaling experiments, latent-space structure analysis, and attention-based saliency.

Results: Finetuned ECG-FM models outperform task-specific baselines in the small-to-medium-scale data regime, exhibit strong label efficiency and cross-dataset generalizability, and achieve high AUROC on salient labels, including atrial fibrillation (0.996) and LVEF $\le 40\%$ (0.929). The pretrained encoder showcases competitive linear probing performance, with functionally discriminative embeddings.

Discussion: Findings indicate that ECG-FM is generalizable, label-efficient, and discriminative for screening, risk stratification, and monitoring. Its representations capture low-level morphology and high-order cardiac semantics, and the pretrained encoder serves as a robust feature-set generator. This work mitigates reliance on large labeled datasets, reduces compute and data requirements, and lowers barriers to reproducibility and cross-study comparison.

Conclusion: ECG-FM is an open, rigorously validated ECG foundation model intended to accelerate transparent, comparable research in the ECG analysis subfield. It is designed for rapid integration and evaluation, especially for delivering practical gains in low-label settings. We release our code, model weights, tutorials, and benchmark at https://github.com/bowang-lab/ECG-FM/.

Objectives: Joint recognition and ICD-10 linking of diagnoses in bilingual, non-standard Spanish and Catalan primary care notes is challenging. We evaluate parameter-efficient fine-tuning (PEFT) techniques as a resource-conscious alternative to full fine-tuning (FFT) for multi-label clinical text classification.

Materials and methods: On a corpus of 21 812 Catalan and Spanish clinical notes from Catalonia, we compared the PEFT techniques LoRA, DoRA, LoHA, LoKR, and QLoRA applied to multilingual transformers (BERT, RoBERTa, DistilBERT, and mDeBERTa).

Results: FFT delivered the best strict Micro-F1 (63.0), but BERT-QLoRA scored 62.2, only 0.8 points lower, while reducing trainable parameters by 67.5% and memory by 33.7%. Training on combined bilingual data consistently improved generalization across individual languages.

Discussion: The small FFT margin was confined to rare labels, indicating limited benefit from updating all parameters. Among PEFT techniques, QLoRA offered the strongest accuracy-efficiency balance; LoRA and DoRA were competitive, whereas LoHA and LoKR incurred larger losses. Adapter rank mattered: ranks below 128 sharply degraded Micro-F1. The substantial memory savings enable deployment on commodity GPUs while delivering performance very close to FFT.

Conclusion: PEFT, particularly QLoRA, supports accurate and memory-efficient joint entity recognition and ICD-10 linking in multilingual, low-resource clinical settings.