Can ChatGPT Help College Instructors Generate High-quality Quiz Questions?

Abstract

ChatGPT is getting increasing attention in both academic and professional settings. Since its release, there has been a discussion on how services such as ChatGPT may change education. Many teachers have shared that they use ChatGPT to help them generate assignment prompts, questions, and lesson plans in various subject areas. Mixed opinions have been shared with regard to the quality of materials created by ChatGPT. While some teachers believe that the materials are of reasonable quality, others worry that ChatGPT may not always generate accurate or reliable information and may reproduce biases and stereotypes that exist in the data it was trained on. In this study, I explore the research question of whether ChatGPT can really replace teachers in generating high-quality assessment questions. Specifically, I compare ChatGPT-generated questions with instructor-written questions that have been used in two classes at a public research University in the US. The preliminary results show that although ChatGPT can produce logically sensible questions, the quality is not always comparable to instructor-written ones. The ChatGPT-generated questions are not specific to student misconceptions and do not align with the learning objectives instructors have in mind, which often leads to such questions being relatively obvious and easy to answer. I further discuss the capabilities and limitations of ChatGPT in generating high-quality assessment questions. This study provides insights into how we may leverage advanced AI tools such as ChatGPT to support education.