Generative AI-Derived Information About Opioid Use Disorder Treatment During Pregnancy: An Exploratory Evaluation of GPT-4's Steerability for Provision of Trustworthy Person-Centered Information.

Abstract

Objective: Increasing engagement in evidence-based treatment for opioid use disorder during pregnancy is pressing. Generative artificial intelligence large language model conversational agents may support clinicians in delivering safe, accurate, and relevant information to this population. The central aim of this study was an exploratory evaluation of the steerability of GPT-4 (Generative Pre-trained Transformer) for the provision of trustworthy treatment-related information to pregnant people with opioid use disorder.

Method: The model was tuned using evidence-based guidelines and tenets of motivational interviewing. A rubric was developed to evaluate the safety, accuracy, and relevance of the tuned model's responses to user messages from the persona of a pregnant woman with an opioid use disorder. Two advanced practice registered nurses with more than 10 years of experience treating people with opioid use disorder independently evaluated the dialogs between the model and persona (n = 30) using the rubric and qualitative methodology.

Results: Responses were rated as safe, accurate, and relevant in 96.7% of cases. Qualitative analysis identified four increasing connection subthemes, including three related to client-centered communication. In 100% of cases, the model identified congruence with opioid use disorder criteria and located the person within the transtheoretical model's stages of change.

Conclusions: The tuned model generated clinically safe, accurate, and relevant responses about opioid use disorder treatment during pregnancy. Consistent with the progression of informatics study typology, before this model could be embedded in an application to allow direct public access, additional lab- and field-based testing is indicated, including with people with this use disorder.