Accuracy of Current Large Language Models and The Retrieval Augmented Generation Model in Determining Dietary Principles in Chronic Kidney Disease.

Abstract

Objective: Large Language Models (LLMs) have emerged as powerful tools with significant potential for quickly accessing information in the nutrition and health, as in many fields. Retrieval augmented generation (RAG) has been included among artificial intelligence (AI) powered chatbot structures as a framework developed to increase the accuracy and ability of LLMs. This study aimed to evaluate the accuracy of LLMs (GPT4, Gemini, and Llama) and RAG in determining dietary principles in chronic kidney disease.

Design and methods: The nutrition guideline published by the National Kidney Foundation in 2020 was used as an external information source in developed RAG model. Answers were obtained using 12 medical nutritional therapy prompts for CKD by four chatbots. The accuracy of the 48 answers generated by the chatbots was evaluated with a 5-point Likert scale.

Results: The results showed that Gemini and RAG had the highest accuracy scores (median:4.0), followed by GPT4 (median: 2.5) and Llama (median: 1.5), respectively. When the accuracy scores were examined between the two chatbots, a significant difference was detected between all groups except Gemini and RAG.

Conclusion: These chatbots produced both completely correct answers and false information with potentially harmful clinical outcomes. Customization of LLMs in specific areas such as nutrition or the development of a nutrition-specific RAG framework by improving LLM structures with current guidelines and articles may be an important strategy to increase the accuracy of AI powered chatbots.