Journal of the American Medical Informatics Association最新文献

Objectives: There is rapidly growing interest in learning health systems (LHSs) nationally and globally. While the critical role of informatics is recognized, the informatics community has been relatively slow to formalize LHS as a priority area.

Materials and methods: We compiled results from a short survey of LHS leaders and American Medical Informatics Association (AMIA) members, discussion from an LHS reception at the AMIA annual meeting, and a follow-up survey to inform priorities at the intersection of LHS and informatics.

Results: We present opportunities between informatics and LHS which fell into themes of: Understanding and Context, Shared Resources, Collaboration, Education, Data, Evaluation, and Patient Centeredness. Immediate LHS informatics priorities identified include establishing informatics LHS forum(s), case reports of LHS informatics successes and failures, LHS informatics education resources, and improved understanding of LHS principles in informatics.

Conclusion: Increased informatics and LHS alignment is critical for advancing this transformative national priority.

Objectives: Successful implementation of machine learning-augmented clinical decision support systems (ML-CDSS) in perioperative care requires the prioritization of patient-centric approaches to ensure alignment with societal expectations. We assessed general public and surgical patient attitudes and perspectives on ML-CDSS use in perioperative care.

Materials and methods: A sequential explanatory study was conducted. Stage 1 collected public opinions through a survey. Stage 2 ascertained surgical patients' experiences and attitudes via focus groups and interviews.

Results: For Stage 1, a total of 281 respondents' (140 males [49.8%]) data were considered. Among participants without ML awareness, males were almost three times more likely than females to report more acceptance (OR = 2.97; 95% CI, 1.36-6.49) and embrace (OR = 2.74; 95% CI, 1.23-6.09) of ML-CDSS use by perioperative teams. Males were almost twice as likely as females to report more acceptance across all perioperative phases with ORs ranging from 1.71 to 2.07. In Stage 2, insights from 10 surgical patients revealed unanimous agreement that ML-CDSS should primarily serve a supportive function. The pre- and post-operative phases were identified explicitly as forums where ML-CDSS can enhance care delivery. Patients requested for education on ML-CDSS's role in their care to be disseminated by surgeons across multiple platforms.

Discussion and conclusion: The general public and surgical patients are receptive to ML-CDSS use throughout their perioperative care provided its role is auxiliary to perioperative teams. However, the integration of ML-CDSS into perioperative workflows presents unique challenges for healthcare settings. Insights from this study can inform strategies to support large-scale implementation and adoption of ML-CDSS by patients in all perioperative phases. Key strategies to promote the feasibility and acceptability of ML-CDSS include clinician-led discussions about ML-CDSS's role in perioperative care, established metrics to evaluate the clinical utility of ML-CDSS, and patient education.

Background: Acute hepatic porphyria (AHP) is a group of rare but treatable conditions associated with diagnostic delays of 15 years on average. The advent of electronic health records (EHR) data and machine learning (ML) may improve the timely recognition of rare diseases like AHP. However, prediction models can be difficult to train given the limited case numbers, unstructured EHR data, and selection biases intrinsic to healthcare delivery. We sought to train and characterize models for identifying patients with AHP.

Methods: This diagnostic study used structured and notes-based EHR data from 2 centers at the University of California, UCSF (2012-2022) and UCLA (2019-2022). The data were split into 2 cohorts (referral and diagnosis) and used to develop models that predict (1) who will be referred for testing of acute porphyria, among those who presented with abdominal pain (a cardinal symptom of AHP), and (2) who will test positive, among those referred. The referral cohort consisted of 747 patients referred for testing and 99 849 contemporaneous patients who were not. The diagnosis cohort consisted of 72 confirmed AHP cases and 347 patients who tested negative. The case cohort was 81% female and 6-75 years old at the time of diagnosis. Candidate models used a range of architectures. Feature selection was semi-automated and incorporated publicly available data from knowledge graphs. Our primary outcome was the F-score on an outcome-stratified test set.

Results: The best center-specific referral models achieved an F-score of 86%-91%. The best diagnosis model achieved an F-score of 92%. To further test our model, we contacted 372 current patients who lack an AHP diagnosis but were predicted by our models as potentially having it (≥10% probability of referral, ≥50% of testing positive). However, we were only able to recruit 10 of these patients for biochemical testing, all of whom were negative. Nonetheless, post hoc evaluations suggested that these models could identify 71% of cases earlier than their diagnosis date, saving 1.2 years.

Conclusions: ML can reduce diagnostic delays in AHP and other rare diseases. Robust recruitment strategies and multicenter coordination will be needed to validate these models before they can be deployed.

Objective: To understand barriers to obtaining and using interoperable information at US hospitals.

Materials and methods: Using 2023 nationally representative survey data on US hospitals (N = 2420), we examined major and minor barriers to exchanging information with other organizations, and how barriers vary by hospital characteristics and methods used to obtain information. Using a series of regression models, we examined how hospital experiences with barriers relate to routine use of information at responding hospitals.

Results: In 2023, most hospitals experienced at least one minor (81%) or major (62%) barrier to exchange, with the most common major barriers relating to different vendors and exchange partners' capabilities. Higher-resourced hospitals and those often using network-based exchange tended to experience more minor barriers whereas lower-resourced hospitals and those often using mail/fax or direct access to outside electronic health records experienced more major barriers. In multivariate regression, hospitals indicating "Patient matching" and "Costs to exchange" were a major or minor barrier had the strongest independent negative association with the likelihood of reporting providers at their hospital frequently use information from outside organizations.

Discussion: Despite progress in interoperable exchange, various barriers remain. The prevalence of barriers varied by hospital type and methods used, with barriers more often preventing exchange for lower-resourced hospitals and those using outdated exchange methods.

Conclusion: While several technical and policy efforts are underway to address prevalent barriers, it will be important to monitor whether efforts are successful in ensuring information from outside organizations can be seamlessly exchanged and used to inform patient care.

Objective: To quantify how many patient scheduled hours would result in a 40-h work week (PSH40) for ambulatory physicians and to determine how PSH40 varies by specialty and practice type.

Methods: We calculated PSH40 for 186 188 ambulatory physicians across 395 organizations from November 2021 through April 2022 stratified by specialty.

Results: Median PSH40 for the sample was 33.2 h (IQR: 28.7-36.5). PSH40 was lowest in infectious disease (26.2, IQR: 21.6-31.1), geriatrics (27.2, IQR: 21.5-32.0) and hematology (28.6, IQR: 23.6-32.6) and highest in plastic surgery (35.7, IQR: 32.8-37.7), pain medicine (35.8, IQR: 32.6-37.9) and sports medicine (36.0, IQR: 33.3-38.1).

Discussion: Health system leaders and physicians will benefit from data driven and transparent discussions about work hour expectations. The PSH40 measure can also be used to quantify the impact of variations in the clinical care environment on the in-person ambulatory patient care time available to physicians.

Conclusions: PSH40 is a novel measure that can be generated from vendor-derived metrics and used by operational leaders to inform work expectations. It can also support research into the impact of changes in the care environment on physicians' workload and capacity.

Objective: We proposed adopting billing models for secure messaging (SM) telehealth services that move beyond time-based metrics, focusing on the complexity and clinical expertise involved in patient care.

Materials and methods: We trained 8 classification machine learning (ML) models using providers' electronic health record (EHR) audit log data for patient-initiated non-urgent messages. Mixed effect modeling (MEM) analyzed significance.

Results: Accuracy and area under the receiver operating characteristics curve scores generally exceeded 0.85, demonstrating robust performance. MEM showed that knowledge domains significantly influenced SM billing, explaining nearly 40% of the variance.

Discussion: This study demonstrates that ML models using EHR audit log data can improve and predict billing in SM telehealth services, supporting billing models that reflect clinical complexity and expertise rather than time-based metrics.

Conclusion: Our research highlights the need for SM billing models beyond time-based metrics, using EHR audit log data to capture the true value of clinical work.

Objectives: Traditional methods for medical device post-market surveillance often fail to accurately account for operator learning effects, leading to biased assessments of device safety. These methods struggle with non-linearity, complex learning curves, and time-varying covariates, such as physician experience. To address these limitations, we sought to develop a machine learning (ML) framework to detect and adjust for operator learning effects.

Materials and methods: A gradient-boosted decision tree ML method was used to analyze synthetic datasets that replicate the complexity of clinical scenarios involving high-risk medical devices. We designed this process to detect learning effects using a risk-adjusted cumulative sum method, quantify the excess adverse event rate attributable to operator inexperience, and adjust for these alongside patient factors in evaluating device safety signals. To maintain integrity, we employed blinding between data generation and analysis teams. Synthetic data used underlying distributions and patient feature correlations based on clinical data from the Department of Veterans Affairs between 2005 and 2012. We generated 2494 synthetic datasets with widely varying characteristics including number of patient features, operators and institutions, and the operator learning form. Each dataset contained a hypothetical study device, Device B, and a reference device, Device A. We evaluated accuracy in identifying learning effects and identifying and estimating the strength of the device safety signal. Our approach also evaluated different clinically relevant thresholds for safety signal detection.

Results: Our framework accurately identified the presence or absence of learning effects in 93.6% of datasets and correctly determined device safety signals in 93.4% of cases. The estimated device odds ratios' 95% confidence intervals were accurately aligned with the specified ratios in 94.7% of datasets. In contrast, a comparative model excluding operator learning effects significantly underperformed in detecting device signals and in accuracy. Notably, our framework achieved 100% specificity for clinically relevant safety signal thresholds, although sensitivity varied with the threshold applied.

Discussion: A machine learning framework, tailored for the complexities of post-market device evaluation, may provide superior performance compared to standard parametric techniques when operator learning is present.

Conclusion: Demonstrating the capacity of ML to overcome complex evaluative challenges, our framework addresses the limitations of traditional statistical methods in current post-market surveillance processes. By offering a reliable means to detect and adjust for learning effects, it may significantly improve medical device safety evaluation.

Objective: Evaluate popular explanation methods using heatmap visualizations to explain the predictions of deep neural networks for electrocardiogram (ECG) analysis and provide recommendations for selection of explanations methods.

Materials and methods: A residual deep neural network was trained on ECGs to predict intervals and amplitudes. Nine commonly used explanation methods (Saliency, Deconvolution, Guided backpropagation, Gradient SHAP, SmoothGrad, Input × gradient, DeepLIFT, Integrated gradients, GradCAM) were qualitatively evaluated by medical experts and objectively evaluated using a perturbation-based method.

Results: No single explanation method consistently outperformed the other methods, but some methods were clearly inferior. We found considerable disagreement between the human expert evaluation and the objective evaluation by perturbation.

Discussion: The best explanation method depended on the ECG measure. To ensure that future explanations of deep neural networks for medical data analyses are useful to medical experts, data scientists developing new explanation methods should collaborate tightly with domain experts. Because there is no explanation method that performs best in all use cases, several methods should be applied.

Conclusion: Several explanation methods should be used to determine the most suitable approach.

Objectives: Social support (SS) and social isolation (SI) are social determinants of health (SDOH) associated with psychiatric outcomes. In electronic health records (EHRs), individual-level SS/SI is typically documented in narrative clinical notes rather than as structured coded data. Natural language processing (NLP) algorithms can automate the otherwise labor-intensive process of extraction of such information.

Materials and methods: Psychiatric encounter notes from Mount Sinai Health System (MSHS, n = 300) and Weill Cornell Medicine (WCM, n = 225) were annotated to create a gold-standard corpus. A rule-based system (RBS) involving lexicons and a large language model (LLM) using FLAN-T5-XL were developed to identify mentions of SS and SI and their subcategories (eg, social network, instrumental support, and loneliness).

Results: For extracting SS/SI, the RBS obtained higher macroaveraged F1-scores than the LLM at both MSHS (0.89 versus 0.65) and WCM (0.85 versus 0.82). For extracting the subcategories, the RBS also outperformed the LLM at both MSHS (0.90 versus 0.62) and WCM (0.82 versus 0.81).

Discussion and conclusion: Unexpectedly, the RBS outperformed the LLMs across all metrics. An intensive review demonstrates that this finding is due to the divergent approach taken by the RBS and LLM. The RBS was designed and refined to follow the same specific rules as the gold-standard annotations. Conversely, the LLM was more inclusive with categorization and conformed to common English-language understanding. Both approaches offer advantages, although additional replication studies are warranted.