Utility of Generative Artificial Intelligence for Patient Care Counseling for Mandibular Fractures.

Abstract

Objective: To determine the readability and accuracy of information regarding mandible fractures generated by Chat Generative Pre-trained Transformer (ChatGPT) versions 3.5 and 4o.

Background: Patients are increasingly turning to generative artificial intelligence to answer medical queries. To date, the accuracy and readability of responses regarding mandible fractures have not been assessed.

Methods: Twenty patient questions regarding mandible fractures were developed by querying AlsoAsked (https://alsoasked.com), SearchResponse (https://searchresponse.io), and Answer the Public (https://answerthepublic.com/). Questions were posed to ChatGPT 3.5 and 4o. Readability was assessed by calculating the Flesch Kincaid Reading Ease, Flesch Kincaid Grade Level, number of sentences, and percentage of complex words. Accuracy was assessed by a board-certified facial plastic and reconstructive otolaryngologist using a 5-point Likert Scale.

Results: No significant differences were observed between the two versions for readability or accuracy. Readability was above recommended levels for patient education materials. Accuracy was low, and a majority of responses were deemed inappropriate for patient use with multiple inaccuracies and/or missing information.

Conclusion: ChatGPT produced responses written at a high level inappropriate for the average patient, in addition to containing several inaccurate statements. Patients and clinicians should be aware of the limitations of generative artificial intelligence when seeking medical information regarding mandible fractures.