Is artificial intelligence successful in the Turkish neurology board exam?

Abstract

Objectives: OpenAI declared that GPT-4 performed better in academic and certain specialty areas. Medical licensing exams assess the clinical competence of doctors. We aimed to investigate for the first time howChatGPT will perform in the Turkish Neurology Proficiency Exam.

Methods: TheGPT-4 version of ChatGPT was used in the study due to the presence ofimage-based questions. The multiple-choice sections of the TurkishNeurology Proficiency Exams conducted by the Turkish NeurologyAssociation (TND) in 2021, 2022 and 2023 were applied to ChatGPT-4.Questions and multiple-choice answers were used in their originalTurkish forms in the official national examination standards.

Results: The success rate in all three exams ranges from 79% to 82%. There were common and different mistakes in the two trials. When the incorrect answers were re-evaluated, the correct answers were obtained.

Conclusions: This is the first study to investigate the performance of ChatGPT on the real Neurology Proficiency Examination. The success rate was shown to be above GPT-3.5. Furthermore, this study showed that translating questions from the original language into English did not affect the performance of GPT-4 in medical licensing exams, unlike GPT-3.5. It is therefore very important that the information obtained is accurate and verifiable. ChatGPT-4's ability to find the correct answer with feedback on questions that it initially answered incorrectly may be due to the model's ability to generate flexible and adaptive answers.These models should be used carefully and consciously, knowing that they will not always give the correct answer.