Evaluating the effectiveness of ChatGPT4 in the diagnosis and workup of dermatologic conditions.

Abstract

ChatGPT is a publicly available chatbot released by OpenAI. Its usefulness in responding to medical questions has been assessed in several specialties, but there is limited literature in dermatology. This study seeks to understand how well ChatGPT4 can provide accurate diagnoses and appropriate workup suggestions for clinical vignettes describing common dermatologic conditions. Ten vignettes were input into ChatGPT4 representing presentations of common dermatologic conditions, written from the perspective of a physician not board-certified in dermatology. ChatGPT4 was asked to identify the top five most likely diagnoses and its recommended workup for each vignette. Responses were assessed quantitatively by calculating the percentage of correct diagnoses, with accurate diagnoses defined by three board-certified dermatologists, and qualitatively using Likert scales describing the accuracy of diagnoses and appropriateness of workups scored by eleven board-certified dermatologists. Overall, 52% of ChatGPT4's diagnoses were accurate and 62% of its recommended workup suggestions were deemed completely correct by board-certified dermatologists. ChatGPT4 was better at recommending an appropriate workup than identifying accurate diagnoses across vignettes. ChatGPT4 was able to accurately diagnose and workup common dermatologic conditions in slightly more than half of cases. ChatGPT4 was better at determining an appropriate workup than an accurate diagnosis.Keywords: artificial intelligence, ChatGPT, dermatology, diagnosis, OpenAI, workup.