Evaluating Large Language Models on their Accuracy and Completeness: Immune Checkpoint Inhibitors and their Ocular Toxicities.

Abstract

Purpose: To analyze the accuracy and thoroughness of 3 large language models (LLMs) to produce information for providers about immune checkpoint inhibitor (ICI) ocular toxicities.

Methods: Eight questions were created about the general definition of checkpoint inhibitors, their mechanism of action, ocular toxicities, and toxicity management. All were inputted into ChatGPT 4.0, Bard, and LLaMA programs. Utilizing the 6-point Likert scale for accuracy and completeness, four ophthalmologists who routinely treat ocular toxicities of immunotherapy agents rated the LLMs answers. ANOVA testing was used to assess significant differences among the three LLMs and a post-hoc pairwise t-test. Fleiss kappa values were calculated to account for interrater variability.

Results: ChatGPT responses were rated with an average of 4.59 for accuracy and 4.09 for completeness; Bard answers were rated 4.59 and 4.19; LLaMA results were rated 4.38 and 4.03. The three LLMs did not significantly differ in accuracy (p=0.47) nor completeness (p=0.86). Fleiss kappa values were found to be poor for both accuracy (-0.03) and completeness (0.01).

Conclusions: All three LLMs provided highly accurate and complete responses to questions centered on ICI inhibitor ocular toxicities and management. Further studies are needed to assess specific ICI agents and the accuracy and completeness of updated versions of LLMs.