Investigating the Role and Impact of Distractors on Parsons Problems in CS1 Assessments

Abstract

In recent years Parsons problems have grown in popularity as both a pedagogical tool and as an assessment item alike. In these problems, students are expected to take existing but jumbled blocks of code and organize them to form a working solution. It is common for these problems to include incorrect blocks of code, typically referred to as "distractors," alongside the correct blocks. However, the utility of these distractors and their impact on a problems difficulty has yet to be thoroughly investigated. This study contributes to filling this gap by comparing performance, time spent, and item discrimination statistics for 32 pairs of Parsons problems from CS1 Python exams and quizzes. Our findings indicate that the inclusion of distractors has a large impact on the amount of time students spend on the questions and a low to moderate impact on score. Additionally, problems without distractors were already found to have high discrimination and including distractors did little to improve their discrimination. These findings suggest that the inclusion of distractors does little to improve the quality of these problems as exam questions but may have a negative impact on students by causing them to spend significantly more time on the problems and reducing the time they have for the rest of the exam.