The Impact of Multiple Choice Question Design on Predictions of Performance

Abstract

Multiple choice questions (MCQs) are a popular question format in introductory programming courses as they are a convenient means to provide scalable assessments. However, with typically only four or five answer options and a single correct answer, MCQs are prone to guessing and may lead students into a false sense of confidence. One approach to mitigate this problem is the use of Multiple-Answer MCQs (MAMCQs), where more than one answer option may be correct. This provides a larger solution space and may help students form more accurate assessments of their knowledge. We explore the use of this question format on an exam in a very large introductory programming course. The exam consisted of both MCQ and MAMCQ sections, and students were invited to predict their scores for each section. In addition, students were asked to report their preference for question format. We found that students over predicted their score on the MCQ section to a greater extent and that these prediction errors were more pronounced amongst less capable students. Interestingly, we found that students did not have a strong preference for MCQs over MAMCQs, and we recommend broader adoption of the latter format.