Effects of Artificial Intelligence-Powered Virtual Agents on Learning Outcomes in Computer-Based Simulations: A Meta-Analysis

Abstract

Computer-based simulations for learning offer affordances for advanced capabilities and expansive possibilities for knowledge construction and skills application. Virtual agents, when powered by artificial intelligence (AI), can be used to scaffold personalized and adaptive learning processes. However, a synthesis or a systematic evaluation of the learning effectiveness of AI-powered virtual agents in computer-based simulations for learning is still lacking. Therefore, this meta-analysis is aimed at evaluating the effects of AI-powered virtual agents in computer-based simulations for learning. The analysis of 49 effect sizes derived from 22 empirical studies suggested a medium positive overall effect, \(\overline{g }=0.43\) , SE = 0.08, 95% C.I. [0.27, 0.59], favoring the use of AI-powered virtual agents over the non-AI-powered virtual agent condition in computer-based simulations for learning. Further, moderator analyses revealed that intervention length, AI technologies, and the representation of AI-powered virtual agents significantly explain the heterogeneity of the overall effects. Conversely, other moderators, including education level, domain, the role of AI-powered virtual agents, the modality of AI-powered virtual agents, and learning environment, appeared to be universally effective among the studies of AI-powered virtual agents in computer-based simulations for learning. Overall, this meta-analysis provides systematic and existing evidence supporting the adoption of AI-powered virtual agents in computer-based simulations for learning. The findings also inform about evidence-based design decisions on the moderators analyzed.