Assessing the performance of an artificial intelligence based chatbot in the differential diagnosis of oral mucosal lesions: clinical validation study.

Abstract

Objectives: Artificial intelligence (AI) is becoming more popular in medicine. The current study aims to investigate, primarily, if an AI-based chatbot, such as ChatGPT, could be a valid tool for assisting in establishing a differential diagnosis of oral mucosal lesions.

Methods: Data was gathered from patients who were referred to our clinic for an oral mucosal biopsy by one oral medicine specialist. Clinical description, differential diagnoses, and final histopathologic diagnoses were retrospectively extracted from patient records. The lesion description was inputted into ChatGPT version 4.0 under a uniform script to generate three differential diagnoses. ChatGPT and an oral medicine specialist's differential diagnosis were compared to the final histopathologic diagnosis.

Results: 100 oral soft tissue lesions were evaluated. A statistically significant correlation was found between the ability of the Chatbot and the Specialist to accurately diagnose the cases (P < 0.001). ChatGPT demonstrated remarkable sensitivity for diagnosing urgent cases, as none of the malignant lesions were missed by the chatbot. At the same time, the specificity of the specialist was higher in cases of malignant lesion diagnosis (p < 0.05). The chatbot performance was reliable in two different events (p < 0.01).

Conclusion: ChatGPT-4 has shown the ability to pinpoint suspicious malignant lesions and suggest an adequate differential diagnosis for soft tissue lesions, in a consistent and repetitive manner.

Clinical relevance: This study serves as a primary insight into the role of AI chatbots, as assisting tools in oral medicine and assesses their clinical capabilities.