Journal of the American Medical Informatics Association最新文献

Objectives: We describe the development and implementation of a system for monitoring patient-reported adverse events and quality of life using electronic Patient Reported Outcome (ePRO) instruments in the I-SPY2 Trial, a phase II clinical trial for locally advanced breast cancer. We describe the administration of technological, workflow, and behavior change interventions and their associated impact on questionnaire completion.

Materials and methods: Using the OpenClinica electronic data capture system, we developed rules-based logic to build automated ePRO surveys, customized to the I-SPY2 treatment schedule. We piloted ePROs at the University of California, San Francisco (UCSF) to optimize workflow in the context of trial treatment scenarios and staggered rollout of the ePRO system to 26 sites to ensure effective implementation of the technology.

Results: Increasing ePRO completion requires workflow solutions and research staff engagement. Over two years, we increased baseline survey completion from 25% to 80%. The majority of patients completed between 30% and 75% of the questionnaires they received, with no statistically significant variation in survey completion by age, race or ethnicity. Patients who completed the screening timepoint questionnaire were significantly more likely to complete more of the surveys they received at later timepoints (mean completion of 74.1% vs 35.5%, P < .0001). Baseline PROMIS social functioning and grade 2 or more PRO-CTCAE interference of Abdominal Pain, Decreased Appetite, Dizziness and Shortness of Breath was associated with lower survey completion rates.

Discussion and conclusion: By implementing ePROs, we have the potential to increase efficiency and accuracy of patient-reported clinical trial data collection, while improving quality of care, patient safety, and health outcomes. Our method is accessible across demographics and facilitates an ease of data collection and sharing across nationwide sites. We identify predictors of decreased completion that can optimize resource allocation by better targeting efforts such as in-person outreach, staff engagement, a robust technical workflow, and increased monitoring to improve overall completion rates.

Trial registration: https://clinicaltrials.gov/study/NCT01042379.

Importance: Scales often arise from multi-item questionnaires, yet commonly face item non-response. Traditional solutions use weighted mean (WMean) from available responses, but potentially overlook missing data intricacies. Advanced methods like multiple imputation (MI) address broader missing data, but demand increased computational resources. Researchers frequently use survey data in the All of Us Research Program (All of Us), and it is imperative to determine if the increased computational burden of employing MI to handle non-response is justifiable.

Objectives: Using the 5-item Physical Activity Neighborhood Environment Scale (PANES) in All of Us, this study assessed the tradeoff between efficacy and computational demands of WMean, MI, and inverse probability weighting (IPW) when dealing with item non-response.

Materials and methods: Synthetic missingness, allowing 1 or more item non-response, was introduced into PANES across 3 missing mechanisms and various missing percentages (10%-50%). Each scenario compared WMean of complete questions, MI, and IPW on bias, variability, coverage probability, and computation time.

Results: All methods showed minimal biases (all <5.5%) for good internal consistency, with WMean suffered most with poor consistency. IPW showed considerable variability with increasing missing percentage. MI required significantly more computational resources, taking >8000 and >100 times longer than WMean and IPW in full data analysis, respectively.

Discussion and conclusion: The marginal performance advantages of MI for item non-response in highly reliable scales do not warrant its escalated cloud computational burden in All of Us, particularly when coupled with computationally demanding post-imputation analyses. Researchers using survey scales with low missingness could utilize WMean to reduce computing burden.

Objectives: The All of Us Research Program is a precision medicine initiative aimed at establishing a vast, diverse biomedical database accessible through a cloud-based data analysis platform, the Researcher Workbench (RW). Our goal was to empower the research community by co-designing the implementation of SAS in the RW alongside researchers to enable broader use of All of Us data.

Materials and methods: Researchers from various fields and with different SAS experience levels participated in co-designing the SAS implementation through user experience interviews.

Results: Feedback and lessons learned from user testing informed the final design of the SAS application.

Discussion: The co-design approach is critical for reducing technical barriers, broadening All of Us data use, and enhancing the user experience for data analysis on the RW.

Conclusion: Our co-design approach successfully tailored the implementation of the SAS application to researchers' needs. This approach may inform future software implementations on the RW.

Objective: The Australian Cancer Atlas (ACA) aims to provide small-area estimates of cancer incidence and survival in Australia to help identify and address geographical health disparities. We report on the 21-month user-centered design study to visualize the data, in particular, the visualization of the estimate uncertainty for multiple audiences.

Materials and methods: The preliminary phases included a scoping study, literature review, and target audience focus groups. Several methods were used to reach the wide target audience. The design and development stage included digital prototyping in parallel with Bayesian model development. Feedback was sought from multiple workshops, audience focus groups, and regular meetings throughout with an expert external advisory group.

Results: The initial scoping identified 4 target audience groups: the general public, researchers, health practitioners, and policy makers. These target groups were consulted throughout the project to ensure the developed model and uncertainty visualizations were effective for communication. In this paper, we detail ACA features and design iterations, including the 3 complementary ways in which uncertainty is communicated: the wave plot, the v-plot, and color transparency.

Discussion: We reflect on the methods, design iterations, decision-making process, and document lessons learned for future atlases.

Conclusion: The ACA has been hugely successful since launching in 2018. It has received over 62 000 individual users from over 100 countries and across all target audiences. It has been replicated in other countries and the second version of the ACA was launched in May 2024. This paper provides rich documentation for future projects.

Objectives: To address the need for interactive visualization tools and databases in characterizing multimorbidity patterns across different populations, we developed the Phenome-wide Multi-Institutional Multimorbidity Explorer (PheMIME). This tool leverages three large-scale EHR systems to facilitate efficient analysis and visualization of disease multimorbidity, aiming to reveal both robust and novel disease associations that are consistent across different systems and to provide insight for enhancing personalized healthcare strategies.

Materials and methods: PheMIME integrates summary statistics from phenome-wide analyses of disease multimorbidities, utilizing data from Vanderbilt University Medical Center, Mass General Brigham, and the UK Biobank. It offers interactive and multifaceted visualizations for exploring multimorbidity. Incorporating an enhanced version of associationSubgraphs, PheMIME also enables dynamic analysis and inference of disease clusters, promoting the discovery of complex multimorbidity patterns. A case study on schizophrenia demonstrates its capability for generating interactive visualizations of multimorbidity networks within and across multiple systems. Additionally, PheMIME supports diverse multimorbidity-based discoveries, detailed further in online case studies.

Results: The PheMIME is accessible at https://prod.tbilab.org/PheMIME/. A comprehensive tutorial and multiple case studies for demonstration are available at https://prod.tbilab.org/PheMIME_supplementary_materials/. The source code can be downloaded from https://github.com/tbilab/PheMIME.

Discussion: PheMIME represents a significant advancement in medical informatics, offering an efficient solution for accessing, analyzing, and interpreting the complex and noisy real-world patient data in electronic health records.

Conclusion: PheMIME provides an extensive multimorbidity knowledge base that consolidates data from three EHR systems, and it is a novel interactive tool designed to analyze and visualize multimorbidities across multiple EHR datasets. It stands out as the first of its kind to offer extensive multimorbidity knowledge integration with substantial support for efficient online analysis and interactive visualization.

Objectives: Large language models (LLMs) have demonstrated remarkable success in natural language processing (NLP) tasks. This study aimed to evaluate their performances on social media-based health-related text classification tasks.

Materials and methods: We benchmarked 1 Support Vector Machine (SVM), 3 supervised pretrained language models (PLMs), and 2 LLMs-based classifiers across 6 text classification tasks. We developed 3 approaches for leveraging LLMs: employing LLMs as zero-shot classifiers, using LLMs as data annotators, and utilizing LLMs with few-shot examples for data augmentation.

Results: Across all tasks, the mean (SD) F1 score differences for RoBERTa, BERTweet, and SocBERT trained on human-annotated data were 0.24 (±0.10), 0.25 (±0.11), and 0.23 (±0.11), respectively, compared to those trained on the data annotated using GPT3.5, and were 0.16 (±0.07), 0.16 (±0.08), and 0.14 (±0.08) using GPT4, respectively. The GPT3.5 and GPT4 zero-shot classifiers outperformed SVMs in a single task and in 5 out of 6 tasks, respectively. When leveraging LLMs for data augmentation, the RoBERTa models trained on GPT4-augmented data demonstrated superior or comparable performance compared to those trained on human-annotated data alone.

Discussion: The results revealed that using LLM-annotated data only for training supervised classification models was ineffective. However, employing the LLM as a zero-shot classifier exhibited the potential to outperform traditional SVM models and achieved a higher recall than the advanced transformer-based model RoBERTa. Additionally, our results indicated that utilizing GPT3.5 for data augmentation could potentially harm model performance. In contrast, data augmentation with GPT4 demonstrated improved model performances, showcasing the potential of LLMs in reducing the need for extensive training data.

Conclusions: By leveraging the data augmentation strategy, we can harness the power of LLMs to develop smaller, more effective domain-specific NLP models. Using LLM-annotated data without human guidance for training lightweight supervised classification models is an ineffective strategy. However, LLM, as a zero-shot classifier, shows promise in excluding false negatives and potentially reducing the human effort required for data annotation.

Objectives: To demonstrate that 2 popular cohort discovery tools, Leaf and the Shared Health Research Information Network (SHRINE), are readily interoperable. Specifically, we adapted Leaf to interoperate and function as a node in a federated data network that uses SHRINE and dynamically generate queries for heterogeneous data models.

Materials and methods: SHRINE queries are designed to run on the Informatics for Integrating Biology & the Bedside (i2b2) data model. We created functionality in Leaf to interoperate with a SHRINE data network and dynamically translate SHRINE queries to other data models. We randomly selected 500 past queries from the SHRINE-based national Evolve to Next-Gen Accrual to Clinical Trials (ENACT) network for evaluation, and an additional 100 queries to refine and debug Leaf's translation functionality. We created a script for Leaf to convert the terms in the SHRINE queries into equivalent structured query language (SQL) concepts, which were then executed on 2 other data models.

Results and discussion: 91.1% of the generated queries for non-i2b2 models returned counts within 5% (or ±5 patients for counts under 100) of i2b2, with 91.3% recall. Of the 8.9% of queries that exceeded the 5% margin, 77 of 89 (86.5%) were due to errors introduced by the Python script or the extract-transform-load process, which are easily fixed in a production deployment. The remaining errors were due to Leaf's translation function, which was later fixed.

Conclusion: Our results support that cohort discovery applications such as Leaf and SHRINE can interoperate in federated data networks with heterogeneous data models.

Objective: This article outlines a scalable system developed by the All of Us Research Program's Genetic Counseling Resource to vet a large database of healthcare resources for supporting participants with health-related DNA results.

Materials and methods: After a literature review of established evaluation frameworks for health resources, we created SONAR, a 10-item framework and grading scale for health-related participant-facing resources. SONAR was used to review clinical resources that could be shared with participants during genetic counseling.

Results: Application of SONAR shortened resource approval time from 7 days to 1 day. About 256 resources were approved and 8 rejected through SONAR review. Most approved resources were relevant to participants nationwide (60.0%). The most common resource types were related to support groups (20%), cancer care (30.6%), and general educational resources (12.4%). All of Us genetic counselors provided 1161 approved resources during 3005 (38.6%) consults, mainly to local genetic counselors (29.9%), support groups (21.9%), and educational resources (21.0%).

Discussion: SONAR's systematic method simplifies resource vetting for healthcare providers, easing the burden of identifying and evaluating credible resources. Compiling these resources into a user-friendly database allows providers to share these resources efficiently, better equipping participants to complete follow up actions from health-related DNA results.

Conclusion: The All of Us Genetic Counseling Resource connects participants receiving health-related DNA results with relevant follow-up resources on a high-volume, national level. This has been made possible by the creation of a novel resource database and validation system.

Objectives: To assess the use of a co-designed patient-reported outcome (PRO) clinical dashboard and estimate its impact on shared decision-making (SDM) and symptomatology in adults with advanced cancer or chronic kidney disease (CKD).

Materials and methods: We developed a clinical PRO dashboard within the Northwestern Medicine Patient-Reported Outcomes system, enhanced through co-design involving 20 diverse constituents. Using a single-group, pretest-posttest design, we evaluated the dashboard's use among patients with advanced cancer or CKD between June 2020 and January 2022. Eligible patients had a visit with a participating clinician, completed at least two dashboard-eligible visits, and consented to follow-up surveys. PROs were collected 72 h prior to visits, including measures for chronic condition management self-efficacy, health-related quality of life (PROMIS measures), and SDM (collaboRATE). Responses were integrated into the EHR dashboard and accessible to clinicians and patients.

Results: We recruited 157 participants: 66 with advanced cancer and 91 with CKD. There were significant improvements in SDM from baseline, as assessed by collaboRATE scores. The proportion of participants reporting the highest level of SDM on every collaboRATE item increased by 15 percentage points from baseline to 3 months, and 17 points between baseline and 6-month follow-up. Additionally, there was a clinically meaningful decrease in anxiety levels over study period (T-score baseline: 53; 3-month: 52; 6-month: 50; P < .001), with a standardized response mean (SRM) of -0.38 at 6 months.

Discussion: PRO clinical dashboards, developed and shared with patients, may enhance SDM and reduce anxiety among patients with advanced cancer and CKD.