Comparing human-made and AI-generated teaching videos: An experimental study on learning effects

Abstract

In the age of generative AI, can teaching videos be efficiently and effectively generated by large language models? In this study, the authors used generative AI tools to develop four short teaching videos for a management course and then compared them with human-generated videos on the same subjects in an online experiment. In an across-subject experimental design, 447 participants completed two treatment conditions presenting different mixes of AI-generated and human-made videos. The participants were asked to rate their learning experiences after each video and had their learning outcomes tested in a multiple-choice exam at the end of the session (N = 1788 video treatments). The findings show that human-generated videos provided a statistically significant but small advantage to participants in terms of learning experience, indicating that the participants still prefer to be taught by human teachers. However, a comparison of exam results between the experimental groups implies that the participants eventually acquired knowledge about the content to a similar degree. Given these findings and the ease with which AI-generated teaching videos can be created, this study concludes that AI-generated teaching videos will likely proliferate.