Comparing the Perspectives of Generative AI, Mental Health Experts, and the General Public on Schizophrenia Recovery: Case Vignette Study

Abstract

Background: Background: The current paradigm in mental healthcare focuses on clinical recovery and symptom remission. This model’s efficacy is influenced by therapist trust in patient recovery potential and therapeutic relationship depth. Schizophrenia is a chronic illness with severe symptoms where the possibility of recovery is a matter of debate. As artificial intelligence (AI) becomes integrated into the healthcare field, it is important to examine its ability to assess recovery potential in major psychiatric disorders such as schizophrenia. Objective: Objectives: To evaluate the ability of Large Languets Models (LLMs) in comparison to mental health professionals to assess the prognosis of schizophrenia with and without treatments and the long term positive and negative outcomes. Methods: Methods: Vignettes were input to LLMs interfaces and assessed ten times by four AI platforms: ChatGPT-3.5, ChatGPT-4, Google Bard, and Claude. A total of 80 evaluations were collected and benchmarked against existing norms to analyze what mental health professionals (general practitioners, psychiatrists, clinical psychologists and mental health nurses) and the general public think about schizophrenia prognosis with and without treatment and the positive and negative long-term outcomes of schizophrenia interventions. Results: Results: Prognosis with professional help: ChatGPT-3.5 was notably pessimistic, whereas ChatGPT-4, Claude and BARD aligned with professional views but differed from the general public. All LLMs believed untreated schizophrenia would remain static or worsen without professional help. Long-term outcomes: ChatGPT-4 and Claude predicted more negative outcomes than BARD and ChatGPT-3.5. For positive outcomes, ChatGPT-3.5 and Claude were more negative than BARD and ChatGPT-4. Conclusions: Conclusions: The findings that three out of the four LLMs aligned closely with the predictions of mental health professionals when considering the 'with treatment' condition is a demonstration of the potential of this technology in providing professional clinical prognosis. The pessimistic assessment of ChatGPT 3.5 is a disturbing finding since it may reduce the motivation of patients to start or persist with treatment for schizophrenia. Overall, while LLMs hold promise in augmenting healthcare, their application necessitates rigorous validation and a harmonious blend with human expertise.