AMIA Joint Summits on Translational Science proceedings. AMIA Joint Summits on Translational Science最新文献

This study explored the efficacy of electronic phenotyping in data labeling for machine learning with a focus on urinary tract infections (UTIs). We contrasted labels from electronic phenotyping against previously published labels such as urine culture positivity. In comparison, electronic phenotyping showed the potential to enhance specificity in UTI labeling while maintaining similar sensitivity and was easily scaled for application to a large dataset suitable for machine learning, which we used to train and validate a machine learning model. Electronic phenotyping offers a valuable method for machine learning label generation in healthcare, with potential benefits for patient care and antimicrobial stewardship. Further research will expand its application and optimize techniques for increased performance.

Health literacy is crucial to supporting good health and is a major national goal. Audio delivery of information is becoming more popular for informing oneself. In this study, we evaluate the effect of audio enhancements in the form of information emphasis and pauses with health texts of varying difficulty and we measure health information comprehension and retention. We produced audio snippets from difficult and easy text and conducted the study on Amazon Mechanical Turk (AMT). Our findings suggest that emphasis matters for both information comprehension and retention. When there is no added pause, emphasizing significant information can lower the perceived difficulty for difficult and easy texts. Comprehension is higher (54%) with correctly placed emphasis for the difficult texts compared to not adding emphasis (50%). Adding a pause lowers perceived difficulty and can improve retention but adversely affects information comprehension.

This paper addresses the challenge of binary relation classification in biomedical Natural Language Processing (NLP), focusing on diverse domains including gene-disease associations, compound protein interactions, and social determinants of health (SDOH). We evaluate different approaches, including fine-tuning Bidirectional Encoder Representations from Transformers (BERT) models and generative Large Language Models (LLMs), and examine their performance in zero and few-shot settings. We also introduce a novel dataset of biomedical text annotated with social and clinical entities to facilitate research into relation classification. Our results underscore the continued complexity of this task for both humans and models. BERT-based models trained on domain-specific data excelled in certain domains and achieved comparable performance and generalization power to generative LLMs in others. Despite these encouraging results, these models are still far from achieving human-level performance. We also highlight the significance of high-quality training data and domain-specific fine-tuning on the performance of all the considered models.

Access to real-world data streams like electronic medical records (EMRs) has accelerated the development of supervised machine learning (ML) models for clinical applications. However, few studies investigate the differential impact of particular features in the EMR on model performance under temporal dataset shift. To explain how features in the EMR impact models over time, this study aggregates features into feature groups by their source (e.g. medication orders, diagnosis codes and lab results) and feature categories based on their reflection of patient pathophysiology or healthcare processes. We adapt Shapley values to explain feature groups' and feature categories' marginal contribution to initial and sustained model performance. We investigate three standard clinical prediction tasks and find that while feature contributions to initial performance differ across tasks, pathophysiological features help mitigate temporal discrimination deterioration. These results provide interpretable insights on how specific feature groups contribute to model performance and robustness to temporal dataset shift.

Electronic health record (EHR) documentation is a leading reason for clinician burnout. While technology-enabled solutions like virtual and digital scribes aim to improve this, there is limited evidence of their effectiveness and minimal guidance for healthcare systems around solution selection and implementation. A transdisciplinary approach, informed by clinician interviews and other considerations, was used to evaluate and select a virtual scribe solution to pilot in a rapid iterative sprint over 12 weeks. Surveys, interviews, and EHR metadata were analyzed over a staggered 30 day implementation with live and asynchronous virtual scribe solutions. Among 16 pilot clinicians, documentation burden metrics decreased for some but not all. Some clinicians had highly positive comments, and others had concerns regarding scribe training and quality. Our findings demonstrate that virtual scribes may reduce documentation burden for some clinicians and describe a method for a collaborative and iterative technology selection process for digital tools in practice.

Consumer-grade heart rate (HR) sensors are widely used for tracking physical and mental health status. We explore the feasibility of using Polar H10 electrocardiogram (ECG) sensor to detect and predict cigarette smoking events in naturalistic settings with several machine learning approaches. We have collected and analyzed data for 28 participants observed over a two-week period. We found that using bidirectional long short-term memory (BiLSTM) with ECG-derived and GPS location input features yielded the highest mean accuracy of 69% for smoking event detection. For predicting smoking events, the highest accuracy of 67% was achieved using the fine-tuned LSTM approach. We also found a significant correlation between accuracy and the number of smoking events available from each participant. Our findings indicate that both detection and prediction of smoking events are feasible but require an individualized approach to training the models, particularly for prediction.

An important problem impacting healthcare is the lack of available experts. Machine learning (ML) models may help resolve this by aiding in screening and diagnosing patients. However, creating large, representative datasets to train models is expensive. We evaluated large language models (LLMs) for data creation. Using Autism Spectrum Disorders (ASD), we prompted GPT-3.5 and GPT-4 to generate 4,200 synthetic examples of behaviors to augment existing medical observations. Our goal is to label behaviors corresponding to autism criteria and improve model accuracy with synthetic training data. We used a BERT classifier pretrained on biomedical literature to assess differences in performance between models. A random sample (N=140) from the LLM-generated data was also evaluated by a clinician and found to contain 83% correct behavioral example-label pairs. Augmenting the dataset increased recall by 13% but decreased precision by 16%. Future work will investigate how different synthetic data characteristics affect ML outcomes.

Objectives: To automatically populate the case report forms (CRFs) for an international, pragmatic, multifactorial, response-adaptive, Bayesian COVID-19 platform trial.

Methods: The locations of focus included 27 hospitals and 2 large electronic health record (EHR) instances (1 Cerner Millennium and 1 Epic) that are part of the same health system in the United States. This paper describes our efforts to use EHR data to automatically populate four of the trial's forms: baseline, daily, discharge, and response-adaptive randomization.

Results: Between April 2020 and May 2022, 417 patients from the UPMC health system were enrolled in the trial. A MySQL-based extract, transform, and load pipeline automatically populated 499 of 526 CRF variables. The populated forms were statistically and manually reviewed and then reported to the trial's international data coordinating center.

Conclusions: We accomplished automatic population of CRFs in a large platform trial and made recommendations for improving this process for future trials.

Care management is a team-based and patient-centered approach to reduce health risks and improve outcomes for managed populations. Post Discharge Management (PDM) is an important care management program at Elevance Health, which is aimed at reducing 30-day readmission risk for recently discharged patients. The current PDM program suffers from low engagement. When assigning interventions to patients, case managers choose the interventions to be conducted in each call only based on their limited personal experiences. In this work, we use deep learning causal inference to analyze the impact of interventions conducted on the first call on consumer engagement in the PDM program, which provides a reliable reference for case managers to select interventions to promote consumer engagement. With three experiments cross-validating the results, our results show that consumers will engage more in the program if the case manager conducts interventions that require more nurse-patient interactions on the first call. On the other hand, conducting less interactive and more technical interventions on the first call leads to relatively poor consumer engagement. These findings correspond to the clinical sense of experienced nurses and are consistent with previous findings in patient engagement in hospital settings.

We proposed a novel measure, Degree of Connectivity with Integration of Freshness (DCIF), to incorporate information freshness into analysis of online resource networks. We conducted a pilot study to apply this new measure to a dataset of online information resources related to COVID-19 risk assessment. Among the 52 nodes, we recorded statistically significant difference between the numerical values of DCIF and the traditional structural measure Degree of Connectivity (DC). Manual reviews of 18 selected nodes showed that DCIF outperformed DC in 11 of them, suggesting potential promise of the proposed new measure. We finalized the protocol for manual review based on the pilot and started a full-scale study. The proposed new measure has the potential to provide quantitative assessment on information freshness for timely and effective dissemination of clinical evidence. Further research is required to address the limitations of this pilot study and to examine the generalization of the findings.