AMIA Joint Summits on Translational Science proceedings. AMIA Joint Summits on Translational Science最新文献

When evaluating the performance of clinical machine learning models, one must consider the deployment population. When the population of patients with observed labels is only a subset of the deployment population (label selection), standard model performance estimates on the observed population may be misleading. In this study we describe three classes of label selection and simulate five causally distinct scenarios to assess how particular selection mechanisms bias a suite of commonly reported binary machine learning model performance metrics. Simulations reveal that when selection is affected by observed features, naive estimates of model discrimination may be misleading. When selection is affected by labels, naive estimates of calibration fail to reflect reality. We borrow traditional weighting estimators from causal inference literature and find that when selection probabilities are properly specified, they recover full population estimates. We then tackle the real-world task of monitoring the performance of deployed machine learning models whose interactions with clinicians feed-back and affect the selection mechanism of the labels. We train three machine learning models to flag low-yield laboratory diagnostics, and simulate their intended consequence of reducing wasteful laboratory utilization. We find that naive estimates of AUROC on the observed population undershoot actual performance by up to 20%. Such a disparity could be large enough to lead to the wrongful termination of a successful clinical decision support tool. We propose an altered deployment procedure, one that combines injected randomization with traditional weighted estimates, and find it recovers true model performance.

Electronic health record (EHR)-derived data can be linked to geospatially distributed socioeconomic and environmental factors to conduct large-scale epidemiologic studies. Ambient NO2 is a known environmental risk factor for asthma. However, health exposure studies often rely on data from geographically sparse regulatory monitors that may not reflect true individual exposure. We contrasted use of interpolated NO2 regulatory monitor data with raw satellite measurements and satellite-derived ground estimates, building on previous work which has computed improved exposure estimates from remotely sensed data. Raw satellite and satellite-derived ground measurements captured spatial variation missed by interpolated ground monitor measurements. Multivariable analyses comparing these three NO2 measurement approaches (interpolated monitor, raw satellite, and satellite-derived) revealed a positive relationship between exposure and asthma exacerbations for both satellite measurements. Exposure-outcome relationships using the interpolated monitor NO2 were inconsistent with known relationships to asthma, suggesting that interpolated monitor data might yield misleading results in small region studies.

The evidence is growing that machine and deep learning methods can learn the subtle differences between the language produced by people with various forms of cognitive impairment such as dementia and cognitively healthy individuals. Valuable public data repositories such as TalkBank have made it possible for researchers in the computational community to join forces and learn from each other to make significant advances in this area. However, due to variability in approaches and data selection strategies used by various researchers, results obtained by different groups have been difficult to compare directly. In this paper, we present TRESTLE (Toolkit for Reproducible Execution of Speech Text and Language Experiments), an open source platform that focuses on two datasets from the TalkBank repository with dementia detection as an illustrative domain. Successfully deployed in the hackallenge (Hackathon/Challenge) of the International Workshop on Health Intelligence at AAAI 2022, TRESTLE provides a precise digital blueprint of the data pre-processing and selection strategies that can be reused via TRESTLE by other researchers seeking comparable results with their peers and current state-of-the-art (SOTA) approaches.

Participant recruitment continues to be a challenge to the success of randomized controlled trials, resulting in increased costs, extended trial timelines and delayed treatment availability. Literature provides evidence that study design features (e.g., trial phase, study site involvement) and trial sponsor are significantly associated with recruitment success. Principal investigators oversee the conduct of clinical trials, including recruitment. Through a cross-sectional survey and a thematic analysis of free-text responses, we assessed the perceptions of sixteen principal investigators regarding success factors for participant recruitment. Study site involvement and funding source do not necessarily make recruitment easier or more challenging from the perspective of the principal investigators. The most commonly used recruitment strategies are also the most effort inefficient (e.g., in-person recruitment, reviewing the electronic medical records for prescreening). Finally, we recommended actionable steps, such as improving staff support and leveraging informatics-driven approaches, to allow clinical researchers to enhance participant recruitment.

Kinship relationship estimation plays a significant role in today's genome studies. Since genetic data are mostly stored and protected in different silos, retrieving the desirable kinship relationships across federated data warehouses is a non-trivial problem. The ability to identify and connect related individuals is important for both research and clinical applications. In this work, we propose a new privacy-preserving kinship relationship estimation framework: Incremental Update Kinship Identification (INK). The proposed framework includes three key components that allow us to control the balance between privacy and accuracy (of kinship estimation): an incremental process coupled with the use of auxiliary information and informative scores. Our empirical evaluation shows that INK can achieve higher kinship identification correctness while exposing fewer genetic markers.

Alzheimer's Disease (AD) is a highly heritable neurodegenerative disorder characterized by memory impairments. Understanding how genetic factors contribute to AD pathology may inform interventions to slow or prevent the progression of AD. We performed stratified genetic analyses of 1,574 Alzheimer's Disease Neuroimaging Initiative (ADNI) participants to examine associations between levels of quantitative traits (QT's) and future diagnosis. The Chow test was employed to determine if an individual's genetic profile affects identified predictive relationships between QT's and future diagnosis. Our chow test analysis discovered that cognitive and PET-based biomarkers differentially predicted future diagnosis when stratifying on allelic dosage of AD loci. Post-hoc bootstrapped and association analyses of biomarkers confirmed differential effects, emphasizing the necessity of stratified models to realize individualized AD diagnosis prediction. This novel application of the Chow test allows for the quantification and direct comparison of genetic-based differences. Our findings, as well as the identified QT-future diagnosis relationships, warrant future investigation from a biological context.

Hotspotting may prevent high healthcare costs surrounding a minority of patients when void of issues such as availability, completeness, and accessibility of information in electronic health records (EHRs). We performed a descriptive study using Barnes-Jewish Hospital patients to assess the availability and accessibility of information that can predict negative outcomes. Manual electronic chart review produced descriptive statistics for a sample of 100 High Resource and 100 Control patient records. The majority of cases were not predictive. Predictive information and their sources were inconsistent. Certain types of patients were more predictive than others, albeit a small percentage of the total. Among the largest and most predictive groups was the most difficult to classify, "Other." These findings were expected and consistent with previous studies but contrast with approaches for attempting prediction such as hotspotting. Further studies may provide solutions to the problems and limitations identified in this study.