Enhancing Adverse Event Reporting With Clinical Language Models: Inpatient Falls.

Abstract

Aims: To develop a method for computationally detecting fall events using clinical language models to complement existing self-reporting mechanisms.

Design: Retrospective observational study.

Methods: Text data were collected from the unstructured nursing notes of three hospitals' electronic health records and the Korean national patient safety reports, totalling 34,480 records covering the period from January 2015 to December 2019. Note-level labelling was conducted by two researchers with 95% agreement. Preprocessing data anonymisation and English translation were followed by semantic validation. Five language models based on pretrained Bidirectional Encoder Representations from Transformers (BERT) and Generative Pretrained Transformer (GPT)-4 with prompt programming were explored. Model performance was assessed using F measurements. Error analysis was conducted for the GPT-4 results.

Results: Fine-tuned BERT models with the English data set outperformed GPT-4, with Bio+Clinical BERT achieving the highest F1 score of 0.98. Fine-tuned Korean BERT with the Korean data set also reached an F1 score of 0.98, while GPT-4 achieved a competitive F1 score of 0.94. GPT-4 with prompt programming showed much higher F1 scores than GPT-4 with a standardised prompt for the English data set (0.85 vs. 0.39) and the Korean data set (0.94 vs. 0.03). The error analysis identified that the common misclassification patterns included fall history and homonyms, causing false positives and implicit expressions and missing contextual information, causing false negatives.

Conclusion: The clinical language model approach, if used alongside the existing self-reporting, promises to increase the chance of identifying the majority of factual falls without the need for additional chart reviews.

Impact: Inpatient falls are often underreported, with up to 91% of incidents missed in self-reports. Using language models, we identified a significant portion of these unreported falls, improving the accuracy of adverse event tracking while reducing the self-reporting burden on nurses.

Patient or public contribution: Not applicable.