Chatbots in education: Hype or help? A meta-analysis

Abstract

Artificial intelligence integration in education, primarily through chatbots, has emerged as a potential solution to address the challenges of catering to students' diverse learning backgrounds. This meta-analysis examined chatbot effectiveness in education, driven by amplified interest since ChatGPT's introduction in 2022. Adhering to PRISMA guidelines, we analyzed 135 effect sizes from 62 studies across diverse educational contexts. Initial results revealed a large positive effect of chatbots on learning performance. However, after controlling for publication bias, we found a significant small to moderate effect. Interaction mode, field of study, intervention duration, and educational level emerged as significant moderators. Text-based interactions, STEM disciplines, longer interventions, and lower educational levels showed the most substantial chatbot effectiveness in enhancing learning performance. Neither chatbot type nor cultural differences significantly influenced learning performance. While our findings support chatbots' potential to enhance performance, they underscore the necessity for targeted implementation when integrating these technologies into learning environments.

Educational relevance statement

Our meta-analysis reveals that chatbots have a small to moderate statistically significant positive effect on student learning performance. While chatbots show promise in enhancing education, particularly in STEM fields, text-based interactions, and for longer-term interventions, their impact may be overestimated due to publication bias. These findings underscore the need for targeted chatbot integration in education. Further research is essential as AI continues to evolve to maximize chatbots' educational benefits.