Development and evaluation of a model to identify publications on the clinical impact of pharmacist interventions

Abstract

Background

Pharmacists are increasingly involved in patient care. Pharmacy practice research helps them identify the most clinically meaningful interventions. However, the lack of a widely accepted controlled vocabulary in this field hinders the discovery of this literature.

Objective

To compare the performance of a machine learning model with manual literature searches in identifying potentially relevant publications on the clinical impact of pharmacist interventions. To describe the dataset that was built.

Methods

An automated PubMed search was performed weekly starting in November 2021. Titles and abstracts were retrieved and independently evaluated by two reviewers to select potentially relevant publications on the clinical impact of pharmacists. A Cohen's kappa score was calculated. Data was collected during an 11-month period to train a machine learning model. It was evaluated prospectively during a 5-month period (predictions were collected without being shown to the reviewers). The performance of the model was compared with manual searches (positive predictive value [PPV] and sensitivity).

Results

A transformers-based model was selected. During the prospective evaluation period, 114/1631 (7 %) publications met selection criteria. If the model had been used, 1273/1631 (78 %) would not have needed review. Only 3/114 (3 %) would have been incorrectly excluded. The model showed a PPV of 0.310 and a sensitivity of 0.974. The best manual search showed a PPV of 0.046 and a sensitivity of 0.711. On December 12, 2023, the dataset contained 8607 publications, of which 544 (6 %) met the criteria. The kappa between reviewers was 0.786. The dataset and the model were used to develop a website and a newsletter to share publications (https://impactpharmacy.net).

Conclusion

A machine learning model was developed and performs better than manual PubMed searches to identify potentially relevant publications. It represents a considerable workload reduction. This tool can assist pharmacists and other stakeholders in finding evidence that support pharmacists' interventions.