Journal of the American Medical Informatics Association最新文献

Objectives: Post-discharge adverse events (AEs) are common and heralded by new and worsening symptoms (NWS). We evaluated the effect of electronic health record (EHR)-integrated digital tools designed to promote quality and safety in hospitalized patients on NWS and AEs after discharge.

Materials and methods: Adult general medicine patients at a community hospital were enrolled. We implemented a dashboard which clinicians used to assess safety risks during interdisciplinary rounds. Post-implementation patients were randomized to complete a discharge checklist whose responses were incorporated into the dashboard. Outcomes were assessed using EHR review and 30-day call data adjudicated by 2 clinicians and analyzed using Poisson regression. We conducted comparisons of each exposure on post-discharge outcomes and used selected variables and NWS as independent predictors to model post-discharge AEs using multivariable logistic regression.

Results: A total of 260 patients (122 pre, 71 post [dashboard], 67 post [dashboard plus discharge checklist]) enrolled. The adjusted incidence rate ratios (aIRR) for NWS and AEs were unchanged in the post- compared to pre-implementation period. For patient-reported NWS, aIRR was non-significantly higher for dashboard plus discharge checklist compared to dashboard participants (1.23 [0.97,1.56], P = .08). For post-implementation patients with an AE, aIRR for duration of injury (>1 week) was significantly lower for dashboard plus discharge checklist compared to dashboard participants (0 [0,0.53], P < .01). In multivariable models, certain patient-reported NWS were associated with AEs (3.76 [1.89,7.82], P < .01).

Discussion: While significant reductions in post-discharge AEs were not observed, checklist participants experiencing a post-discharge AE were more likely to report NWS and had a shorter duration of injury.

Conclusion: Interventions designed to prompt patients to report NWS may facilitate earlier detection of AEs after discharge.

Clinicaltrials.gov: NCT05232656.

Importance: Reinforcement learning (RL) represents a pivotal avenue within natural language processing (NLP), offering a potent mechanism for acquiring optimal strategies in task completion. This literature review studies various NLP applications where RL has demonstrated efficacy, with notable applications in healthcare settings.

Objectives: To systematically explore the applications of RL in NLP, focusing on its effectiveness in acquiring optimal strategies, particularly in healthcare settings, and provide a comprehensive understanding of RL's potential in NLP tasks.

Materials and methods: Adhering to the PRISMA guidelines, an exhaustive literature review was conducted to identify instances where RL has exhibited success in NLP applications, encompassing dialogue systems, machine translation, question-answering, text summarization, and information extraction. Our methodological approach involves closely examining the technical aspects of RL methodologies employed in these applications, analyzing algorithms, states, rewards, actions, datasets, and encoder-decoder architectures.

Results: The review of 93 papers yields insights into RL algorithms, prevalent techniques, emergent trends, and the fusion of RL methods in NLP healthcare applications. It clarifies the strategic approaches employed, datasets utilized, and the dynamic terrain of RL-NLP systems, thereby offering a roadmap for research and development in RL and machine learning techniques in healthcare. The review also addresses ethical concerns to ensure equity, transparency, and accountability in the evolution and application of RL-based NLP technologies, particularly within sensitive domains such as healthcare.

Discussion: The findings underscore the promising role of RL in advancing NLP applications, particularly in healthcare, where its potential to optimize decision-making and enhance patient outcomes is significant. However, the ethical challenges and technical complexities associated with RL demand careful consideration and ongoing research to ensure responsible and effective implementation.

Conclusions: By systematically exploring RL's applications in NLP and providing insights into technical analysis, ethical implications, and potential advancements, this review contributes to a deeper understanding of RL's role for language processing.

Objective: This study aims to conduct a systematic review and meta-analysis of the diagnostic accuracy of deep learning (DL) using speech samples in depression.

Materials and methods: This review included studies reporting diagnostic results of DL algorithms in depression using speech data, published from inception to January 31, 2024, on PubMed, Medline, Embase, PsycINFO, Scopus, IEEE, and Web of Science databases. Pooled accuracy, sensitivity, and specificity were obtained by random-effect models. The diagnostic Precision Study Quality Assessment Tool (QUADAS-2) was used to assess the risk of bias.

Results: A total of 25 studies met the inclusion criteria and 8 of them were used in the meta-analysis. The pooled estimates of accuracy, specificity, and sensitivity for depression detection models were 0.87 (95% CI, 0.81-0.93), 0.85 (95% CI, 0.78-0.91), and 0.82 (95% CI, 0.71-0.94), respectively. When stratified by model structure, the highest pooled diagnostic accuracy was 0.89 (95% CI, 0.81-0.97) in the handcrafted group.

Discussion: To our knowledge, our study is the first meta-analysis on the diagnostic performance of DL for depression detection from speech samples. All studies included in the meta-analysis used convolutional neural network (CNN) models, posing problems in deciphering the performance of other DL algorithms. The handcrafted model performed better than the end-to-end model in speech depression detection.

Conclusions: The application of DL in speech provided a useful tool for depression detection. CNN models with handcrafted acoustic features could help to improve the diagnostic performance.

Protocol registration: The study protocol was registered on PROSPERO (CRD42023423603).

Objective: We proposed adopting billing models for secure messaging (SM) telehealth services that move beyond time-based metrics, focusing on the complexity and clinical expertise involved in patient care.

Materials and methods: We trained 8 classification machine learning (ML) models using providers' electronic health record (EHR) audit log data for patient-initiated non-urgent messages. Mixed effect modeling (MEM) analyzed significance.

Results: Accuracy and area under the receiver operating characteristics curve scores generally exceeded 0.85, demonstrating robust performance. MEM showed that knowledge domains significantly influenced SM billing, explaining nearly 40% of the variance.

Discussion: This study demonstrates that ML models using EHR audit log data can improve and predict billing in SM telehealth services, supporting billing models that reflect clinical complexity and expertise rather than time-based metrics.

Conclusion: Our research highlights the need for SM billing models beyond time-based metrics, using EHR audit log data to capture the true value of clinical work.

Significance: Research on the conditions under which electronic cigarette (EC) use produces a net reduction in the population harm attributable to combusted cigarette (CC) use requires the triangulation of information from cohort(s) of smokers, non-smokers, EC users, and dual-users of all varieties.

Materials and methods: This project utilizes data from the All of Us Research Program to contrast a panel of wellness and disease-risk indicators across a range of self-reported tobacco-use profiles, including smokers, current, and former EC users. This article focuses on the tobacco use history and current tobacco use status among All of Us participants enrolled between May 2017 and February 2023 (Registered Controlled Tier Curated Data Repository [CDR] v7).

Results: The present analytic sample included an unweighted total of N = 412 211 individuals with information on ever-use of both CC and EC. Among them, 155 901 individuals have a history of CC use, with 65 206 identified as current smokers. EC usage is reported by 64 002 individuals, with 16 619 being current users. Model predicted analyses identified distinct patterns in CC and EC usage across demographic and socioeconomic variables, with younger ages favoring ECs.

Discussion: Age was observed to significantly affect EC usage, and gender differences reveal that males were significantly more likely to use CC and/or EC than females or African Americans of any gender. Higher educational achievement and income were associated with lower use of both CC and EC, while lower levels of mental health were observed to increase the likelihood of using CC and EC products.

Conclusion: Findings suggest the potential for the All of Us Research Program for investigation of causal factors driving both behavioral use transitions and cessation outcomes.

Objectives: Disparity in kidney transplant access has been demonstrated by a disproportionately low rate of kidney transplantation in socioeconomically disadvantaged patients. However, the information is not from national representative populations with end-stage kidney disease (ESKD). We aim to examine whether socioeconomic disparity for kidney transplant access exists by utilizing data from the All of Us Research Program.

Materials and methods: We analyzed data of adult ESKD patients using the All of Us Researcher Workbench. The association of socioeconomic data including types of health insurance, levels of education, and household incomes with kidney transplant access was evaluated by multivariable logistic regression analysis adjusted by baseline demographic, medical comorbidities, and behavioral information.

Results: Among 4078 adults with ESKD, mean diagnosis age was 54 and 51.64% were male. The majority had Medicare (39.6%), were non-graduate college (75.79%), and earned $10 000-24 999 annual income (20.16%). After adjusting for potential confounders, insurance status emerged as a significant predictor of kidney transplant access. Individuals covered by Medicaid (adjusted odds ratio [AOR] 0.45; 95% confidence interval [CI], 0.35-0.58; P-value < .001) or uninsured (AOR 0.21; 95% CI, 0.12-0.37; P-value < .001) exhibited lower odds of transplantation compared to those with private insurance.

Discussion/conclusion: Our findings reveal the influence of insurance status and socioeconomic factors on access to kidney transplantation among ESKD patients. Addressing these disparities through expanded insurance coverage and improved healthcare access is vital for promoting equitable treatment and enhancing health outcomes in vulnerable populations.

Objectives: To evaluate the proficiency of a HIPAA-compliant version of GPT-4 in identifying actionable, incidental findings from unstructured radiology reports of Emergency Department patients. To assess appropriateness of artificial intelligence (AI)-generated, patient-facing summaries of these findings.

Materials and methods: Radiology reports extracted from the electronic health record of a large academic medical center were manually reviewed to identify non-emergent, incidental findings with high likelihood of requiring follow-up, further sub-stratified as "definitely actionable" (DA) or "possibly actionable-clinical correlation" (PA-CC). Instruction prompts to GPT-4 were developed and iteratively optimized using a validation set of 50 reports. The optimized prompt was then applied to a test set of 430 unseen reports. GPT-4 performance was primarily graded on accuracy identifying either DA or PA-CC findings, then secondarily for DA findings alone. Outputs were reviewed for hallucinations. AI-generated patient-facing summaries were assessed for appropriateness via Likert scale.

Results: For the primary outcome (DA or PA-CC), GPT-4 achieved 99.3% recall, 73.6% precision, and 84.5% F-1. For the secondary outcome (DA only), GPT-4 demonstrated 95.2% recall, 77.3% precision, and 85.3% F-1. No findings were "hallucinated" outright. However, 2.8% of cases included generated text about recommendations that were inferred without specific reference. The majority of True Positive AI-generated summaries required no or minor revision.

Conclusion: GPT-4 demonstrates proficiency in detecting actionable, incidental findings after refined instruction prompting. AI-generated patient instructions were most often appropriate, but rarely included inferred recommendations. While this technology shows promise to augment diagnostics, active clinician oversight via "human-in-the-loop" workflows remains critical for clinical implementation.

Objective: This paper aims to address the challenges in abstract screening within systematic reviews (SR) by leveraging the zero-shot capabilities of large language models (LLMs).

Methods: We employ LLM to prioritize candidate studies by aligning abstracts with the selection criteria outlined in an SR protocol. Abstract screening was transformed into a novel question-answering (QA) framework, treating each selection criterion as a question addressed by LLM. The framework involves breaking down the selection criteria into multiple questions, properly prompting LLM to answer each question, scoring and re-ranking each answer, and combining the responses to make nuanced inclusion or exclusion decisions.

Results and discussion: Large-scale validation was performed on the benchmark of CLEF eHealth 2019 Task 2: Technology-Assisted Reviews in Empirical Medicine. Focusing on GPT-3.5 as a case study, the proposed QA framework consistently exhibited a clear advantage over traditional information retrieval approaches and bespoke BERT-family models that were fine-tuned for prioritizing candidate studies (ie, from the BERT to PubMedBERT) across 31 datasets of 4 categories of SRs, underscoring their high potential in facilitating abstract screening. The experiments also showcased the viability of using selection criteria as a query for reference prioritization. The experiments also showcased the viability of the framework using different LLMs.

Conclusion: Investigation justified the indispensable value of leveraging selection criteria to improve the performance of automated abstract screening. LLMs demonstrated proficiency in prioritizing candidate studies for abstract screening using the proposed QA framework. Significant performance improvements were obtained by re-ranking answers using the semantic alignment between abstracts and selection criteria. This further highlighted the pertinence of utilizing selection criteria to enhance abstract screening.

Objectives: To enhance the performance of large language models (LLMs) in biomedical natural language processing (BioNLP) by introducing a domain-specific instruction dataset and examining its impact when combined with multi-task learning principles.

Materials and methods: We created the BioInstruct, comprising 25 005 instructions to instruction-tune LLMs (LLaMA 1 and 2, 7B and 13B version). The instructions were created by prompting the GPT-4 language model with 3-seed samples randomly drawn from an 80 human curated instructions. We employed Low-Rank Adaptation (LoRA) for parameter-efficient fine-tuning. We then evaluated these instruction-tuned LLMs on several BioNLP tasks, which can be grouped into 3 major categories: question answering (QA), information extraction (IE), and text generation (GEN). We also examined whether categories (eg, QA, IE, and generation) of instructions impact model performance.

Results and discussion: Comparing with LLMs without instruction-tuned, our instruction-tuned LLMs demonstrated marked performance gains: 17.3% in QA on average accuracy metric, 5.7% in IE on average F1 metric, and 96% in Generation tasks on average GPT-4 score metric. Our 7B-parameter instruction-tuned LLaMA 1 model was competitive or even surpassed other LLMs in the biomedical domain that were also fine-tuned from LLaMA 1 with vast domain-specific data or a variety of tasks. Our results also show that the performance gain is significantly higher when instruction fine-tuning is conducted with closely related tasks. Our findings align with the observations of multi-task learning, suggesting the synergies between 2 tasks.

Conclusion: The BioInstruct dataset serves as a valuable resource and instruction tuned LLMs lead to the best performing BioNLP applications.