Is ChatGPT a reliable tool in Autoimmune Hepatitis?

Abstract

Background and aims: Artificial intelligence-based chatbots offer a potential avenue for delivering personalized counselling to Autoimmune Hepatitis (AIH) patients. We assessed accuracy, completeness, comprehensiveness and safety of ChatGPT-4 responses to 12 inquiries out of a pool of 40 questions posed by four AIH patients.

Methods: Questions were categorized into three areas: Diagnosis(1-3), Quality of Life(4-8) and Medical treatment(9-12). 11 Key Opinion Leaders (KOLs) evaluated responses using a Likert scale with 6 points for accuracy, 5 points for safety and 3 points for completeness and comprehensiveness.

Results: Median scores for accuracy, completeness, comprehensiveness and safety were 5(4-6), 2 (2-2) and 3 (2-3); no domain exhibited superior evaluation. Post-diagnosis follow-up question was the trickiest with low accuracy and completeness but safe and comprehensive features. Agreement among KOLs (Fleiss's Kappa statistics) was slight for accuracy (0.05) but poor for the remaining features (-0.05, -0.06 and -0,02, respectively).

Conclusions: Chatbots show good comprehensibility but lack reliability. Further studies are needed to integrate Chat-GPT within clinical practice.