ChatGPT-4o's Performance in Brain Tumor Diagnosis and MRI Findings: A Comparative Analysis with Radiologists

Abstract

Rationale and Objectives

To evaluate the accuracy of ChatGPT-4o in identifying magnetic resonance imaging (MRI) findings and diagnosing brain tumors by comparing its performance with that of experienced radiologists.

Materials and Methods

This retrospective study included 46 patients with pathologically confirmed brain tumors who underwent preoperative MRI between January 2021 and October 2024. Two experienced radiologists and ChatGPT 4o independently evaluated the anonymized MRI images. Eight questions focusing on MRI sequences, lesion characteristics, and diagnoses were answered. ChatGPT-4o's responses were compared to those of the radiologists and the pathology outcomes. Statistical analyses were performed, which included accuracy, sensitivity, specificity, and the McNemar test, with p <0.05 considered to indicate a statistically significant difference.

Results

ChatGPT-4o successfully identified 44 of the 46 (95.7%) lesions; it achieved 88.3% accuracy in identifying MRI sequences, 81% in perilesional edema, 79.5% in signal characteristics, and 82.2% in contrast enhancement. However, its accuracy in localizing lesions was 53.6% and that in distinguishing extra-axial from intra-axial lesions was 26.3%. As such, ChatGPT-4o achieved success rates of 56.8% and 29.5% for differential diagnoses and most likely diagnoses when compared to 93.2–90.9% and 70.5–65.9% for radiologists, respectively (p <0.005).

Conclusion

ChatGPT-4o demonstrated high accuracy in identifying certain MRI features but underperformed in diagnostic tasks in comparison with the radiologists. Despite its current limitations, future updates and advancements have the potential to enable large language models to facilitate diagnosis and offer a reliable second opinion to radiologists.