Recommendations for initial diabetic retinopathy screening of diabetic patients using large language model-based artificial intelligence in real-life case scenarios.

Abstract

Purpose: To study the role of artificial intelligence (AI) to identify key risk factors for diabetic retinopathy (DR) screening and develop recommendations based on clinician and large language model (LLM) based AI platform opinions for newly detected diabetes mellitus (DM) cases.

Methods: Five clinicians and three AI applications were given 20 AI-generated hypothetical case scenarios to assess DR screening timing. We calculated inter-rater agreements between clinicians, AI-platforms, and the "majority clinician response" (defined as the maximum number of identical responses provided by the clinicians) and "majority AI-platform" (defined as the maximum number of identical responses among the 3 distinct AI). Scoring was used to identify risk factors of different severity. Three, two, and one points were given to risk factors requiring screening immediately, within a year, and within five years, respectively. After calculating a cumulative screening score, categories were assigned.

Results: Clinicians, AI platforms, and the "majority clinician response" and "majority AI response" had fair inter-rater reliability (k value: 0.21-0.40). Uncontrolled DM and systemic co-morbidities required immediate screening, while family history of DM and a co-existing pregnancy required screening within a year. The absence of these risk factors required screening within 5 years of DM diagnosis. Screening scores in this study were between 0 and 10. Cases with screening scores of 0-2 needed screening within 5 years, 3-5 within 1 year, and 6-12 immediately.

Conclusion: Based on the findings of this study, AI could play a critical role in DR screening of newly diagnosed DM patients by developing a novel DR screening score. Future studies would be required to validate the DR screening score before it could be used as a reference in real-life clinical situations.

Clinical trial registration: Not applicable.