Applied Clinical Informatics最新文献

Background: The integration of large language models (LLMs) into clinical diagnostics presents significant challenges regarding their accuracy and reliability.

Objective: This study aimed to evaluate the performance of DeepSeek R1, an open-source reasoning model, alongside two other LLMs, GPT-4.1 and Claude 3.5 Sonnet, across multiple-choice clinical cases.

Methods: A dataset of complex medical cases representative of real-world clinical practice was selected. For efficiency, models were accessed via application programming interfaces (APIs) and assessed using standardized prompts and a predefined evaluation protocol.

Results: The models demonstrated an overall accuracy of 77.1%, with GPT-4 producing the fewest errors and Claude 3.5 the most. The reproducibility analysis indicated that the tests were very repeatable: DeepSeek (100%), GPT-4.1 (97.5%), and Claude 3.5 Sonnet (92%).

Conclusions: While LLMs show promise for enhancing diagnostics, ongoing scrutiny is required to address error rates and validate standard medical answers. Given the limited dataset and prompting protocol, findings should not be interpreted as broader equivalence in real‑world clinical reasoning. This study demonstrates the need for robust evaluation standards, attention to error rates, and further research.

Background: Push notifications are a common method of clinical communication in inpatient settings, yet their volume and delivery patterns have not been described. Alert fatigue has been well described in healthcare, and push notifications may be a new contributor.

Objective: To characterize the volume, type, and temporal distribution of push notifications received by hospitalists across distinct clinical roles in a large academic health system.

Methods: We conducted a cross-sectional analysis of electronic health record (EHR) audit log data from June 1, 2024, to June 1, 2025, at a large academic health system using Epic (Verona, WI) EHR. All push notifications received by attending hospitalists were extracted, categorized (secure message, results, other), and summarized by hour, hospitalist role, and device type.

Results: Ninety-seven hospitalists received 1,114,657 push notifications over one year, with a median of 11 (3-24) notifications per hour. Rounding hospitalists received 9 (7-12) notifications per patient per working day. Secure message notifications accounted for the majority and result-related notifications comprised only 2.2% of notifications. Notifications peaked midday and were received throughout the day, including outside of scheduled shift times.

Conclusions: Hospitalists are exposed to a high volume of push notifications, which may contribute to alert fatigue and ultimately impact patient safety and clinician wellbeing. System-level efforts to prioritize clinically meaningful notifications, refine notification settings, and enhance secure-messaging infrastructure are needed to protect clinician attention and support patient safety.

Background and significance: Getting patients out of intensive care units (ICUs) is a major goal for acute care clinicians, as prolonged stays increase the risk of complications and strain critical resources such as staff, equipment, and beds. The ICU Liberation bundle or the ABCDEF (A-F) care bundle is an evidence-based framework for improving outcomes in critically ill patients by addressing pain, sedation, delirium, mobility, and family engagement. However, variability in documentation and lack of standardized data elements hinder effective implementation and evaluation of adherence to bundle components.

Objectives: This study aims to characterize data elements of the A-F liberation bundle using a large, single-center critical care database and to develop standardized bundle cards that map bundle components to controlled vocabularies.

Methods: We conducted a retrospective analysis of data elements related to A-F bundle using the MIMIC-IV database. Clinical concepts were mapped to standardized vocabularies and aligned with the OMOP common data model. Bundle cards were developed for each component to provide structured, accessible documentation of assessment tools, adherence criteria, and terminology mappings.

Results: Pain assessments were documented in over 11,000 patients, with a median of 23 assessments per day. Sedation levels for nearly 59,000 patients were evaluated, with 37.7% meeting Society of Critical Care Medicine (SCCM) adherence criteria. Delirium assessments followed standardized protocols incorporating RASS and CAM-ICU scores. Components E and F lacked formal compliance specifications; bundle cards for these components identified key activities and highlighted gaps in standardized vocabularies. Adherence analyses revealed variability likely due to non-standardized documentation practices.

Conclusion: We developed and validated six ICU Liberation Bundle cards that map bundle components to standardized vocabularies and common data models, enabling retrospective adherence evaluation in real-world data. These information resources promote consistent documentation, support interoperability, and provide a foundation for prospective monitoring to enhance bundle implementation in critical care.

Drive to Zero is a mobile health application (app) designed to identify and retain people with HIV (PWH) who have experienced challenges with achieving or maintaining viral suppression. The app targets PWH who have lacked documented HIV care in the past months and are experiencing medication adherence barriers. Features include an interactive chat for communicating with the study team and access to educational resources to support care engagement and health management.This usability study aimed to assess the Drive to Zero app's ease of use and interface design through expert heuristic evaluation and end-user testing.Usability was evaluated through two approaches: heuristic evaluations conducted by five informatics experts following Nielsen's usability principles, and end-user testing with 20 PWH using the validated Post-Study System Usability Questionnaire and qualitative interviews to collect feedback on app functionality and user experience.Heuristic experts and end-users demonstrated satisfaction with the app's appearance, reporting that it has a simple and intuitive interface for identifying and retaining PWH, which will assist them with study engagement and ultimately reengage with HIV care. However, participants highlighted areas needing improvement, suggesting better accessibility of "home" and "help" buttons to improve user control and a more detailed explanation of the incentive program to enhance user engagement and retention.Usability evaluations provided valuable insights into the Drive to Zero app's design. Areas for improvement were enhancing user controls and improving the readability of the incentive program. These findings will guide iterative refinements, ensuring that future versions of the app improve the usability and acceptability of its target audience.

Abandoned medication orders-those initiated but not signed-represent a potential safety risk and an indicator of electronic health record (EHR) inefficiency. This study explores inpatient medication abandonment across two large tertiary healthcare systems using different EHRs.Silent alerts were deployed to identify abandoned orders at Site 1 (June 2018-May 2019) and Site 2 (July 2020-May 2023). At Site 1, alerts triggered on all inpatient medication orders. At Site 2, alerts were part of a broader study implementing indication alerts; only orders for study medications triggered alerts. An abandoned order was defined as an order initiated but not signed within 24 hours of initiation. We calculated abandonment rates and rates of reorders, and performed regression to examine the association between abandonment and clinician, patient, and order characteristics. Exponential models were fit to characterize the chronology of reordering.Among 6.8 million medication orders, abandonment rates were 11.2% at Site 1 and 25.0% at Site 2. Due to fundamental differences in alert configuration and order capture, no direct statistical comparison of abandonment rates between the two sites was conducted. Over half of abandoned orders were reordered within 24 hours (65.3% at Site 1; 54.2% at Site 2). The chronology of reordering was similar at both institutions. Attendings, the most senior clinicians, had the lowest rates of abandonment. Abandonment rates decreased as clinicians placed more orders, but rose as clinicians ordered on more unique patients. Abandonments were higher when ordering for children compared with adults.Order abandonment is common and varies by patient's age, clinician type, and workload. Abandonment rates declined as house staff providers advanced in training, signifying clinical experience plays a role. Frequent reordering suggests that workflow interruptions or modifications, rather than intentional medication cancellation, may lead to a significant proportion of abandonments. Similarity in the timing of reordering between healthcare systems suggest common reordering processes across sites. Our findings demonstrate significant order abandonment rates, with the potential to use abandonment as a metric to improve computerized provider order entry (CPOE) functionality, clinicians' workflows, and patient safety.

Identifying patient portals (PP) activation disparities, especially in electronic health record (EHR) activation workflows, can help facilitate equitable health care access.Our study aimed to assess whether the parent/guardian's preferred language was associated with being offered, activating, and using the PP and the methods used to offer activation codes.This retrospective cohort study examined PP offer, activation, and usage rates at a large freestanding children's hospital. Patients <12 years old with ambulatory visits from July 1, 2022, to June 30, 2023, without prior active proxy PP accounts were included. The primary independent variable was the self-reported parent/guardian preferred language (English/Spanish). Outcomes included the probability of being offered, overall and by specific offer methods, activation, and usage. Zou's modified multivariate Poisson regression models examined the association between preferred language and offer/activate/use status.Among 39,578 patients, 85.1% were patients with English as preferred language (PEPL) and 14.9% had Spanish as preferred language (PSPL). PSPL had a lower probability of being offered (adjusted relative risk ratio [aRR]: 0.65, 95% confidence interval [CI]: 0.63-0.67), activated (aRR: 0.72, 95% CI: 0.70-0.75), and used (aRR: 0.68, 95% CI: 0.65-0.72) a PP compared to PEPL. Specifically, PSPL had a lower probability of activating if ever offered via instant activation (aRR: 0.72, 95% CI: 0.69-0.75), parent/guardian with existing account (aRR: 0.73, 95% CI: 0.69-0.76), and had equal probability of activating if ever offered via letter (aRR: 0.42, 95% CI: 0.19-0.94) and clinician-assisted method (aRR: 0.99, 95% CI: 0.86-1.16), compared to PEPL.PSPL at a large, free-standing pediatric health system had a lower probability of PP offer, activation, and usage than PEPL. Activation methods were not universally effective across language groups, emphasizing the need for equitable workflow optimization. This study highlights an approach to analyzing health disparities in activation workflows to inform targeted interventions to improve equitable PP access.

The digital transformation of healthcare is reshaping care delivery among healthcare professionals, requiring nurses to develop digital competencies. These competencies are essential but often underdeveloped due to limited training and resources. Global initiatives emphasize integrating these competencies into nursing education, necessitating valid instruments to assess them.This systematic review aims to identify instruments measuring digital competence in nursing and to assess their measurement properties.This review was registered in PROSPERO (identifier: CRD42024522349) and conducted according to PRISMA guidelines. A systematic search was performed in CINAHL, PubMed/MEDLINE, and Scopus on instruments assessing digital competencies in nursing and reporting measurement properties. Measurement properties and their methodological quality were assessed using the COSMIN criteria, and the overall quality of the evidence was graded using a modified GRADE approach.A total of 27 instruments were identified, relating to three interconnected constructs: nursing informatics, digital health, and information and communication technology. Based on their measurement properties, the instruments were categorized into three groups (A, B, C) following the COSMIN methodology to support recommendations for use. Six instruments were classified under category A (recommended for use): the DigiHealthCom and DigiComInf instruments, the Turkish version of TANIC, the short version of ITASH, the Digital Competence Questionnaire, and the 30-item Arabic version of SANICS. Twenty instruments were categorized under category B (potentially recommendable, but further validation is needed). One instrument was placed in category C (not recommended for use).As digital competence becomes an increasing priority in education and public health, valid and reliable instruments are essential for assessing and monitoring these competencies. Such instruments support the identification of training needs, the evaluation of educational outcomes, and the integration of digital skills into nursing curricula and clinical practice, ultimately strengthening the digital readiness of the nursing workforce.

Interruptive clinical decision support (CDS) alerts are intended to standardize patient care and prevent harm. However, failures can occur even in organizations with mature CDS governance and advanced analytics. These breakdowns, marked by excessive firings, workflow disruption, and clinician dissatisfaction, can provide insights into systemic weaknesses in CDS design, testing, and monitoring processes.This study aimed to examine a CDS alert malfunction as a lens for identifying system-level gaps and propose strategies to strengthen resilience in CDS operations.A retrospective analysis was conducted on an interruptive alert that was developed through a phased, multistakeholder, committee-driven process, but was removed within 10 days due to poor performance, revealing gaps that persisted despite established governance.The alert fired 1,866 times in 5 days, with a 91% dismissal rate and reports of workflow disruption. Feedback indicated provider frustration and concern for malfunction. Analysis revealed gaps in end-user engagement, testing rigor, committee reviews, and monitoring practices.CDS failures can serve as catalysts for system improvement. This case highlights actionable lessons, such as operationalizing user-centered design, clarifying testing expectations, and distributing monitoring responsibilities, to enhance CDS reliability. Even well-established governance structures must be continuously evaluated and adapted to keep pace with evolving CDS technologies, and such investments position organizations to maintain responsive, sustainable systems aligned with high-quality care.

Claude Opus 4 is a large language model (LLM) that features improved reasoning capabilities and broader contextual understanding compared to earlier versions. Despite the growing use of LLM systems for seeking medical information, structured and simulation-based evaluations of Claude Opus 4's capabilities in diabetes management remain limited, particularly across domains such as patient education, clinical reasoning, and emotional support.This study aimed to conduct a baseline evaluation of Claude Opus 4's performance across key domains of diabetes care (i.e., patient education, clinical reasoning, and emotional support), and to identify preliminary insights that can inform future, evidence-based integration strategies.A three-step evaluation was conducted: (1) 30 diabetes management questions assessed using expert endocrinologist evaluation, (2) five fictional diabetes cases evaluated for clinical decision-making, and (3) emotional support responses assessed for appropriateness and empathy. Three expert endocrinologists graded responses according to American Diabetes Association guidelines.Claude Opus 4 achieved 80% accuracy in general diabetes knowledge, with high response reproducibility (96.7%), indicating baseline rather than clinically adequate performance. Clinical case evaluations showed moderate utility (mean expert rating = 4.4/7), while emotional-support assessments yielded high scores for empathy (6.2/7) and appropriateness (6.0/7). These findings suggest that although the model demonstrates promising informational and emotional-support capabilities, its current performance remains insufficient for autonomous clinical use and should be viewed as preliminary evidence to guide future, patient-inclusive validation studies.Although Claude Opus 4 demonstrates preliminary findings suggesting potential applications in diabetes care, education, and emotional support, this baseline assessment using fictional cases underscores the need for real-world validation with clinical data to determine true clinical utility and patient-centered impact. This simulation-based evaluation also offers practical lessons learned for researchers designing future LLM assessments, highlighting the need for mixed expert-patient panels, contextual validation, and person-centered metrics beyond numerical accuracy.

Clinical documentation improvement is critical for pediatric care, yet leveraging electronic health record (EHR) tools for this population is not well established. We aimed to adapt and implement a real-time, automated documentation assistance tool (AutoDx) to decrease clinical documentation integrity (CDI) coding queries and improve perceived ease of practice for pediatric inpatient providers.In this quality improvement study at an urban academic pediatric hospital, we adapted and implemented AutoDx for pediatric use by developing and validating pediatric-specific logic rules to alert providers to potential diagnoses based on EHR data. The primary outcome was the rate of CDI queries per 1,000 discharges for targeted diagnoses, aiming for a 50% reduction over a 5-month implementation period compared with a 12-month baseline. Secondary outcomes included provider-surveyed ease of practice, with a goal of a 25% improvement, and tool uptake.The aggregate rate of targeted CDI queries decreased by 58% postimplementation, from 80.7 to 33.9 per 1,000 discharges (p < 0.001). Moreover, analysis by interrupted time series demonstrated an immediate 45.5% reduction in the rate of coding queries (p = 0.028) following the implementation of the tool. The rate of queries for nontargeted diagnoses remained unchanged. Tool adoption increased steadily throughout the study period. While provider-reported time spent on queries did not significantly decrease, a majority of survey respondents (59%) perceived receiving fewer queries, and 46% agreed the tool made it easier to provide quality care.Implementation of a real-time, automated documentation support tool in a pediatric inpatient setting significantly reduced CDI coding queries for targeted diagnoses. Despite a "task substitution" effect where perceived workload did not decrease, the tool improved perceived ease of practice, demonstrating that targeted EHR interventions can enhance documentation accuracy and efficiency in pediatrics.