JAMIA Open最新文献

Objective: Digital dashboards are used to monitor patients and improve inpatient outcomes in hospital settings. A systematic review assessed the impact of dashboards across five outcomes of hospital mortality, hospital length of stay (LOS), economic impacts, harms, and patient and carer satisfaction.

Materials and methods: Nine databases were searched from inception to May 2024. Studies were included if they reported primary quantitative research on dashboard interventions in hospital settings, were in English, and measured effectiveness for patients, caregivers, healthcare professionals or services. Data synthesis was performed via narrative review. Risk of bias was measured using Cochrane ROBINS-I and RoB 2.

Results: We identified 5755 articles, and 70 met inclusion criteria. Of 20 findings reporting mortality (16 studies), five reported a decrease, whilst the majority (n = 15) found no significant change. LOS was reported across 43 findings (31 studies), with 28 reporting a reduction, an increase in five, and ten reporting no change. Of 21 findings (from 16 studies) reporting on harms, increases were observed in six, decreases in four, and no change in 11. Economic impacts were reported in 34 findings (31 studies), with the majority demonstrating reduced costs (n = 29), an increase in one, and no change in four. Eight findings (eight studies) reported on patient and carer satisfaction with care, with the majority (n = 6) demonstrating increased satisfaction, and two reporting no change.

Discussion: Hospital dashboards do appear associated with either no change or a reduction in mortality, reduced costs, reduced LOS, and improved patient and caregiver satisfaction with care. Association with harms was equivocal.

Conclusion: While there is evidence of potential benefits, actual impacts of hospital digital dashboard will likely be dependent on multiple local factors such as workflow integration.

Objective: To develop and evaluate machine learning (ML) models for predicting length of stay (LOS) in elective spine surgery, with a focus on the benefits of temporal modeling and model interpretability.

Materials and methods: We compared traditional ML models (eg, Linear Regression, Random Forest, Support Vector Machine [SVM], and XGBoost) with our developed model, SurgeryLSTM, a masked bidirectional long short-term memory (BiLSTM) with an attention, using structured perioperative electronic health records (EHR) data. Performance was evaluated using the coefficient of determination (R ²), and key predictors were identified using explainable AI.

Results: SurgeryLSTM achieved the highest predictive accuracy (R ² = 0.86), outperforming XGBoost (R ² = 0.85) and baseline models. The attention mechanism improved interpretability by dynamically identifying influential temporal segments within preoperative clinical sequences, allowing clinicians to trace which events or features most contributed to each LOS prediction. Key predictors of LOS included bone disorder, chronic kidney disease, and lumbar fusion identified as the most impactful predictors of LOS.

Discussion: Temporal modeling with attention mechanisms significantly improves LOS prediction by capturing the sequential nature of patient data. Unlike static models, SurgeryLSTM provides both higher accuracy and greater interpretability, which are critical for clinical adoption. These results highlight the potential of integrating attention-based temporal models into hospital planning workflows.

Conclusion: SurgeryLSTM presents an effective and interpretable AI solution for LOS prediction in elective spine surgery. Our findings support the integration of temporal, explainable ML approaches into clinical decision support systems to enhance discharge readiness and individualized patient care.

Objectives: To develop and test a novel machine learning approach for monitoring impact of computerized clinical decision support (CDS) tools on clinicians' electronic health record (EHR) activities.

Materials and methods: Our CDS monitoring approach leverages topic modeling, a latent-variable statistical machine learning method, to infer health providers' EHR activities from EHR audit logs. We applied this approach to monitor the impact of a tobacco cessation support CDS tool newly implemented in 5 cancer clinics (2018-2021). We trained the topic model on EHR audit log data from 3445 encounters (pre-CDS-implementation: 1734, post-CDS-implementation: 1711) for patients with active smoking status. The number of topics was automatically determined based on within-topic coherence and across-topic divergence, and the identified topics were assigned clinically relevant EHR activity labels by 4 domain experts.

Results: The topic model identified 2 distinct activities focusing on CDS (act on CDS, bypass/postpone CDS), 2 activities related to CDS (review patient records and address alerts, use note templates and acknowledge the completion of CDS), 6 related to accessing (access patient station) and reviewing patient data (external records, synopsis data, snapshot of patient data, problem list/diagnosis/notes, treatment plan), and 4 related to modifying EHR (modify diagnosis/problem lists, document visit with record review, perform administrative activities for visit and billing, and document follow-up care plan). Comparing matched 1-hour after-check-in windows post-implementation (n = 841) versus pre-implementation (n = 841) of CDS, the mean prevalence (expressed as proportions out of 1.0) of providers' EHR-use activity increased on CDS-focused activities (0.073, 95% CI, 0.066-0.079) and CDS-related activities (0.098, 95% CI, 0.089-0.106) and decreased on modifying EHR (-0.113, 95% CI, -0.124 to -0.102) and reviewing patient data (-0.058, 95% CI, -0.072 to -0.044).

Discussion: Our topic model-based CDS monitoring approach can identify shifts in prevalence of EHR-use activities pre-implementation versus post-implementation. This approach can be applied to detect unintended changes in EHR activities on a large population scale following CDS implementation, providing valuable insights to guide focused qualitative investigations for CDS improvement or de-implementation.

Conclusion: Our approach offers a scalable, data-driven framework for evaluating the real-world impact of EHR-embedded CDS tools. Built on a generic machine learning framework, this approach could be adapted to explore impact of other healthcare quality improvement strategies using EHR-integrated CDS interventions.

Background: While most health-care providers now use electronic health records (EHRs) to document clinical care, many still treat them as digital versions of paper records. As a result, documentation often remains unstructured, with free-text entries in progress notes. This limits the potential for secondary use and analysis, as machine-learning and data analysis algorithms are more effective with structured data.

Objective: This study aims to use advanced artificial intelligence (AI) and natural language processing (NLP) techniques to improve diagnostic information extraction from clinical notes in a periodontal use case. By automating this process, the study seeks to reduce missing data in dental records and minimize the need for extensive manual annotation, a long-standing barrier to widespread NLP deployment in dental data extraction.

Materials and methods: This research utilizes large language models (LLMs), specifically Generative Pretrained Transformer 4, to generate synthetic medical notes for fine-tuning a RoBERTa model. This model was trained to better interpret and process dental language, with particular attention to periodontal diagnoses. Model performance was evaluated by manually reviewing 360 clinical notes randomly selected from each of the participating site's dataset.

Results: The results demonstrated high accuracy of periodontal diagnosis data extraction, with the sites 1 and 2 achieving a weighted average score of 0.97-0.98. This performance held for all dimensions of periodontal diagnosis in terms of stage, grade, and extent.

Discussion: Synthetic data effectively reduced manual annotation needs while preserving model quality. Generalizability across institutions suggests viability for broader adoption, though future work is needed to improve contextual understanding.

Conclusion: The study highlights the potential transformative impact of AI and NLP on health-care research. Most clinical documentation (40%-80%) is free text. Scaling our method could enhance clinical data reuse.

Objectives: Measles continues to pose a serious threat to global public health, fueled by declining vaccination rates, international travel, and persistent immunization gaps. Early outbreak detection and response remain hampered by fragmented surveillance systems, which often lack interoperability and limit data accessibility.

Materials and methods: To address the major limitations of current measles surveillance systems-including data fragmentation and lack of standardization-we developed Measles Tracker, an integrated near-real-time data hub that centralizes and harmonizes measles surveillance data in the United States using publicly available sources. The system aggregates data from multiple layers, including: (1) official reports from public health agencies, (2) epidemiological surveillance bulletins, and (3) outbreak reports, mainly captured through news websites or via news aggregators. The platform architecture implements (1) geospatial normalization of key epidemiological variables (case counts, vaccination coverage, age-stratified incidence) and (2) dynamic visualization interfaces to support coordination of evidence-based response.

Results: Measles Tracker enhances situational awareness by integrating disparate data streams in near real-time, enabling rapid geospatial detection of outbreak clusters, mapping vaccination gaps, and supporting dynamic risk stratification of vulnerable populations. It is intended exclusively as a complementary tool to official public health systems, providing educational and situational awareness without interfering with contact tracing, vaccination, or outbreak control activities.

Conclusions: As a centralized, scalable tool, Measles Tracker advances measles surveillance by leveraging digital epidemiology principles. Future iterations will incorporate additional data streams (eg, climate variables, genomic surveillance) and advanced analytics (eg, machine learning for risk prediction, network models for transmission dynamics) to further optimize outbreak preparedness and resource allocation. This framework underscores the transformative potential of integrated data systems in global measles elimination efforts.

Objective: This study compares the performance of machine learning (ML) models and human experts in mapping unstructured nursing notes to the standardized Nursing Interventions Classification (NIC) system. The aim is to advance automated nursing documentation classification, facilitating cross-facility benchmarking of patient care and organizational outcomes.

Materials and methods: We developed and compared 4 ML models: TF-IDF text-based vectorization, UMLS semantic mapping, fine-tuned GPT-4o mini, and Bio-Clinical BERT. These models were evaluated against classifications provided by 2 expert nurses using a dataset of de-identified home healthcare nursing notes obtained from a Florida, USA-based medical clearinghouse. Model performance was assessed using agreement statistics, precision, recall, F1 scores, and Cohen's Kappa.

Results: Human raters achieved the highest agreement with consensus labels, scoring 0.75 and 0.62, with corresponding F1 scores of 0.61 and 0.45, respectively. In comparison, ML models showed lower performance, with GPT achieving the best among them (agreement: 0.50, F1 score: 0.31). A distribution analysis of NIC categories revealed that ML models performed well in prevalent and clearly defined categories, such as drug management, but struggled with minority classes and context-dependent interventions, like information management.

Discussion: Current ML approaches show promise in supporting clinical classification tasks, but the performance gap in handling complex, context-dependent interventions highlights the need for improved methods that can better capture the nuanced nature of clinical documentation. Future research should focus on developing methods to process clinical terminology and context-specific documentation with greater precision and adaptability.

Conclusion: Current ML models can aid-but not fully replace-human judgment in classifying nuanced nursing interventions.

Objective: The International Classification of Health Interventions (ICHI), currently being developed, seeks to span all sectors of the health system. Our objective was to determine the coverage of the ICHI for hearing interventions commonly delivered to adults with sensorineural hearing loss (SNHL).

Material and methods: A 3-phase content mapping method was used, which included (1) identification of source terms with an expert panel in audiology rehabilitation; (2) 3 coders independently applied the classification to the source terms; and (3) the coders reached a consensus for each intervention and identified reasons for initial discrepancies with options not linked to a specific code were identified.

Results: Nineteen different ICHI Target categories were identified, with 23 different ICHI Action categories and 82% of the means being "Other and unspecified." There was consensus in codes for 54.3% of source terms, with no ICHI code found for 8.5% of source terms. The greatest number of discrepancies arose from the action, followed by the target. Coding discrepancies occurred as a result of misunderstanding of source terms, the clinical use thereof, and difficulty determining the type of Target.

Discussion: Despite its broad scope, ICHI's current framework has gaps in its coverage of audiological interventions, particularly those related to sensorineural hearing loss. Addressing these gaps is crucial for improving global data standardization and facilitating the development of more targeted hearing health policies.

Conclusion: This study makes an important contribution to the further development and refinement of the classification, specifically in the context of hearing healthcare.

Importance: Data governance, the policies, and procedures for managing data, is a critical factor for secondary use of clinical data for research.

Objectives: This paper describes the evolution of an academic health-care organization's data governance for research, development of an external data sharing process, implementation of related processes, continuous improvement, and ongoing observations of data governance maturity.

Materials and methods: The program was designed to improve the access to and sharing of real-world data for research. Using a combination of qualitative and quantitative methods, we evaluated the program's effectiveness.

Results: Our results describe a significant improvement in data accessibility as seen in new data-driven performance indicators and in data understanding indicated by new processes, policies, and strategies.

Discussion: The paper outlines the development of a data governance process at an academic health center to support external data sharing, emphasizing the importance of data literacy, cross-office collaboration, and structured workflows to manage complex review requirements. The formalized process improved data access, identified gaps, and enabled continuous quality improvement, though it introduced new bottlenecks and required navigating multi-office reviews and researcher education.

Conclusion: These findings suggest data governance practices that may apply to other institutions.

Objective: Falls are a leading cause of morbidity and mortality among older adults. Common methods for identifying fall-related ED visits within both claims and electronic health record datasets rely on diagnosis code-based definitions, which underestimate the true prevalence of falls. This study applies a natural language processing (NLP) algorithm to ED provider notes to identify patients presenting due to falls and compares the characteristics of NLP-identified cases to those identified through diagnosis codes to identify the impact of identification strategy.

Materials and methods: This cross-sectional study analyzed ED encounter data from older adult patients who visited an ED between December 2016 and 2020. The NLP algorithm identified falls based on provider notes, searching for keywords related to falls and excluding negated and spurious matches. We also applied common ICD code methods to identify falls.

Results: We processed 50 153 ED encounters and the NLP approach identified 14 604 encounters for patients who fell. Of those, 7086 (49%) were not identified using external cause of morbidity ICD codes. Patients identified by just the NLP algorithm exhibited higher Elixhauser comorbidity scores and increased likelihood of 30-day mortality. Patients identified by NLP algorithm but not ICD codes were more likely to have severe underlying conditions such as sepsis or acute kidney disease rather than traumatic injuries.

Discussion: The NLP algorithm identifies many fall-related visits not identified by traditional methods.

Conclusion: If the causal relationships between falls and comorbid conditions are not considered in NLP algorithms, they can easily identify patients who fell, but the fall was a sequela of underlying medical illness.

Objectives: To develop and apply a reproducible methodology for evaluating generative artificial intelligence (AI) powered systems in health care, addressing the gap between theoretical evaluation frameworks and practical implementation guidance.

Materials and methods: A 5-dimension evaluation framework was developed to assess query comprehension and response helpfulness, correctness, completeness, and potential clinical harm. The framework was applied to evaluate ClinicalKey AI using queries drawn from user logs, a benchmark dataset, and subject matter expert curated queries. Forty-one board-certified physicians and pharmacists were recruited to independently evaluate query-response pairs. An agreement protocol using the mode and modified Delphi method resolved disagreements in evaluation scores.

Results: Of 633 queries, 614 (96.99%) produced evaluable responses, with subject matter experts completing evaluations of 426 query-response pairs. Results demonstrated high rates of response correctness (95.5%) and query comprehension (98.6%), with 94.4% of responses rated as helpful. Two responses (0.47%) received scores indicating potential clinical harm. Pairwise consensus occurred in 60.6% of evaluations, with remaining cases requiring third tie-breaker review.

Discussion: The framework demonstrated effectiveness in quantifying performance through comprehensive evaluation dimensions and structured scoring resolution methods. Key strengths included representative query sampling, standardized rating scales, and robust subject matter expert agreement protocols. Challenges emerged in managing subjective assessments of open-ended responses and achieving consensus on potential harm classification.

Conclusion: This framework offers a reproducible methodology for evaluating health-care generative AI systems, establishing foundational processes that can inform future efforts while supporting the implementation of generative AI applications in clinical settings.