Journal of the American Medical Informatics Association最新文献

Objective: Evaluate popular explanation methods using heatmap visualizations to explain the predictions of deep neural networks for electrocardiogram (ECG) analysis and provide recommendations for selection of explanations methods.

Materials and methods: A residual deep neural network was trained on ECGs to predict intervals and amplitudes. Nine commonly used explanation methods (Saliency, Deconvolution, Guided backpropagation, Gradient SHAP, SmoothGrad, Input × gradient, DeepLIFT, Integrated gradients, GradCAM) were qualitatively evaluated by medical experts and objectively evaluated using a perturbation-based method.

Results: No single explanation method consistently outperformed the other methods, but some methods were clearly inferior. We found considerable disagreement between the human expert evaluation and the objective evaluation by perturbation.

Discussion: The best explanation method depended on the ECG measure. To ensure that future explanations of deep neural networks for medical data analyses are useful to medical experts, data scientists developing new explanation methods should collaborate tightly with domain experts. Because there is no explanation method that performs best in all use cases, several methods should be applied.

Conclusion: Several explanation methods should be used to determine the most suitable approach.

Objectives: We analyzed trends in adoption of advanced patient engagement and clinical data analytics functionalities among critical access hospitals (CAHs) and non-CAHs to assess how historical gaps have changed.

Materials and methods: We used 2014, 2018, and 2023 data from the American Hospital Association Annual Survey IT Supplement to measure differences in adoption rates (ie, the "adoption gap") of patient engagement and clinical data analytics functionalities across CAHs and non-CAHs. We measured changes over time in CAH and non-CAH adoption of 6 "core" clinical data analytics functionalities, 5 "core" patient engagement functionalities, 5 new patient engagement functionalities, and 3 bulk data export use cases. We constructed 2 composite measures for core functionalities and analyzed adoption for other functionalities individually.

Results: Core functionality adoption increased from 21% of CAHs in 2014 to 56% in 2023 for clinical data analytics and 18% to 49% for patient engagement. The CAH adoption gap in both domains narrowed from 2018 to 2023 (both P < .01). More than 90% of all hospitals had adopted viewing and downloading electronic data and clinical notes by 2023. The largest CAH adoption gaps in 2023 were for Fast Healthcare Interoperability Resources (FHIR) bulk export use cases (eg, analytics and reporting: 63% of CAHs, 81% of non-CAHs, P < .001).

Discussion: Adoption of advanced electronic health record functionalities has increased for CAHs and non-CAHs, and some adoption gaps have been closed since 2018. However, CAHs may continue to struggle with clinical data analytics and FHIR-based functionalities.

Conclusion: Some crucial patient engagement functionalities have reached near-universal adoption; however, policymakers should consider programs to support CAHs in closing remaining adoption gaps.

Objectives: To understand how health-related social needs (HRSN) data are collected at US hospitals and implications for use.

Materials and methods: Using 2023 nationally representative survey data on US hospitals (N = 2775), we described hospitals' routine and structured collection and use of HRSN data and examined the relationship between methods of data collection and specific uses. Multivariate logistic regression was used to identify characteristics associated with data collection and use and understand how methods of data collection relate to use.

Results: In 2023, 88% of hospitals collected HRSN data (64% routinely, 72% structured). While hospitals commonly used data for internal purposes (eg, discharge planning, 79%), those that collected data routinely and in a structured format (58%) used data for purposes involving coordination or exchange with other organizations (eg, making referrals, 74%) at higher rates than hospitals that collected data but not routinely or in a non-structured format (eg, 93% vs 67% for referrals, P< .05). In multivariate regression, routine and structured data collection was positively associated with all uses of data examined. Hospital location, ownership, system-affiliation, value-based care participation, and critical access designation were associated with HRSN data collection, but only system-affiliation was consistently (positively) associated with use.

Discussion: While most hospitals screen for social needs, fewer collect data routinely and in a structured format that would facilitate downstream use. Routine and structured data collection was associated with greater use, particularly for secondary purposes.

Conclusion: Routine and structured screening may result in more actionable data that facilitates use for various purposes that support patient care and improve community and population health, indicating the importance of continuing efforts to increase routine screening and standardize HRSN data collection.

Objective: This communication presents the results of defining a tribal health jurisdiction by a combination of tribal affiliation (TA) and case address.

Materials and methods: Through a county-tribal partnership, Geographic Information System (GIS) software and custom code were used to extract tribal data from county data by identifying reservation addresses in county extracts of COVID-19 case records from December 30, 2019, to December 31, 2022 (n = 374 653) and COVID-19 vaccination records from December 1, 2020, to April 18, 2023 (n = 2 355 058).

Results: The tool identified 1.91 times as many case records and 3.76 times as many vaccination records as filtering by TA alone.

Discussion and conclusion: This method of identifying communities by patient address, in combination with TA and enrollment, can help tribal health jurisdictions attain equitable access to public health data, when done in partnership with a data sharing agreement. This methodology has potential applications for other populations underrepresented in public health and clinical research.

Objectives: Traditional methods for medical device post-market surveillance often fail to accurately account for operator learning effects, leading to biased assessments of device safety. These methods struggle with non-linearity, complex learning curves, and time-varying covariates, such as physician experience. To address these limitations, we sought to develop a machine learning (ML) framework to detect and adjust for operator learning effects.

Materials and methods: A gradient-boosted decision tree ML method was used to analyze synthetic datasets that replicate the complexity of clinical scenarios involving high-risk medical devices. We designed this process to detect learning effects using a risk-adjusted cumulative sum method, quantify the excess adverse event rate attributable to operator inexperience, and adjust for these alongside patient factors in evaluating device safety signals. To maintain integrity, we employed blinding between data generation and analysis teams. Synthetic data used underlying distributions and patient feature correlations based on clinical data from the Department of Veterans Affairs between 2005 and 2012. We generated 2494 synthetic datasets with widely varying characteristics including number of patient features, operators and institutions, and the operator learning form. Each dataset contained a hypothetical study device, Device B, and a reference device, Device A. We evaluated accuracy in identifying learning effects and identifying and estimating the strength of the device safety signal. Our approach also evaluated different clinically relevant thresholds for safety signal detection.

Results: Our framework accurately identified the presence or absence of learning effects in 93.6% of datasets and correctly determined device safety signals in 93.4% of cases. The estimated device odds ratios' 95% confidence intervals were accurately aligned with the specified ratios in 94.7% of datasets. In contrast, a comparative model excluding operator learning effects significantly underperformed in detecting device signals and in accuracy. Notably, our framework achieved 100% specificity for clinically relevant safety signal thresholds, although sensitivity varied with the threshold applied.

Discussion: A machine learning framework, tailored for the complexities of post-market device evaluation, may provide superior performance compared to standard parametric techniques when operator learning is present.

Conclusion: Demonstrating the capacity of ML to overcome complex evaluative challenges, our framework addresses the limitations of traditional statistical methods in current post-market surveillance processes. By offering a reliable means to detect and adjust for learning effects, it may significantly improve medical device safety evaluation.

Objective: Federated Learning (FL) enables collaborative model training while keeping data locally. Currently, most FL studies in radiology are conducted in simulated environments due to numerous hurdles impeding its translation into practice. The few existing real-world FL initiatives rarely communicate specific measures taken to overcome these hurdles. To bridge this significant knowledge gap, we propose a comprehensive guide for real-world FL in radiology. Minding efforts to implement real-world FL, there is a lack of comprehensive assessments comparing FL to less complex alternatives in challenging real-world settings, which we address through extensive benchmarking.

Materials and methods: We developed our own FL infrastructure within the German Radiological Cooperative Network (RACOON) and demonstrated its functionality by training FL models on lung pathology segmentation tasks across six university hospitals. Insights gained while establishing our FL initiative and running the extensive benchmark experiments were compiled and categorized into the guide.

Results: The proposed guide outlines essential steps, identified hurdles, and implemented solutions for establishing successful FL initiatives conducting real-world experiments. Our experimental results prove the practical relevance of our guide and show that FL outperforms less complex alternatives in all evaluation scenarios.

Discussion and conclusion: Our findings justify the efforts required to translate FL into real-world applications by demonstrating advantageous performance over alternative approaches. Additionally, they emphasize the importance of strategic organization, robust management of distributed data and infrastructure in real-world settings. With the proposed guide, we are aiming to aid future FL researchers in circumventing pitfalls and accelerating translation of FL into radiological applications.

Objective: The NIH All of Us Research Program aims to advance personalized medicine by not only linking patient records, surveys, and genomic data but also engaging with participants, particularly from groups traditionally underrepresented in biomedical research (UBR). This study details how the dialogue between scientists and community members, including many from communities of color, shaped local research priorities.

Materials and methods: We recruited area quantitative, basic, and clinical scientists as well as community members from our Community and Participant Advisory Boards with a predetermined interest in All of Us research as members of a Special Interest Group (SIG). An expert community engagement scientist facilitated 6 SIG meetings over the year, explicitly fostering openness and flexibility during conversations. We qualitatively analyzed discussions using a social movement framework tailored for community-based participatory research (CBPR) mobilization.

Results: The SIG evolved through CBPR stages of emergence, coalescence, momentum, and maintenance/integration. Researchers prioritized community needs above personal academic interests while community members kept discussions focused on tangible return of value to communities. One key outcome includes SIG-driven shifts in programmatic and research priorities of the All of Us Research Program in Southeastern Wisconsin. One major challenge was building equitable conversations that balanced scientific rigor and community understanding.

Discussion: Our approach allowed for a rich dialogue to emerge. Points of connection and disconnection between community members and scientists offered important guidance for emerging areas of genomic inquiry.

Conclusion: Our study presents a robust foundation for future efforts to engage diverse communities in CBPR, particularly on healthcare concerns affecting UBR communities.

Objectives: Social support (SS) and social isolation (SI) are social determinants of health (SDOH) associated with psychiatric outcomes. In electronic health records (EHRs), individual-level SS/SI is typically documented in narrative clinical notes rather than as structured coded data. Natural language processing (NLP) algorithms can automate the otherwise labor-intensive process of extraction of such information.

Materials and methods: Psychiatric encounter notes from Mount Sinai Health System (MSHS, n = 300) and Weill Cornell Medicine (WCM, n = 225) were annotated to create a gold-standard corpus. A rule-based system (RBS) involving lexicons and a large language model (LLM) using FLAN-T5-XL were developed to identify mentions of SS and SI and their subcategories (eg, social network, instrumental support, and loneliness).

Results: For extracting SS/SI, the RBS obtained higher macroaveraged F1-scores than the LLM at both MSHS (0.89 versus 0.65) and WCM (0.85 versus 0.82). For extracting the subcategories, the RBS also outperformed the LLM at both MSHS (0.90 versus 0.62) and WCM (0.82 versus 0.81).

Discussion and conclusion: Unexpectedly, the RBS outperformed the LLMs across all metrics. An intensive review demonstrates that this finding is due to the divergent approach taken by the RBS and LLM. The RBS was designed and refined to follow the same specific rules as the gold-standard annotations. Conversely, the LLM was more inclusive with categorization and conformed to common English-language understanding. Both approaches offer advantages, although additional replication studies are warranted.