Evaluation of Submission Limits and Regression Penalties to Improve Student Behavior with Automatic Assessment Systems

Objectives. Automatic assessment systems are widely used to provide rapid feedback for students and reduce grading time. Despite the benefits of increased efficiency and improved pedagogical outcomes, an ongoing challenge is mitigating poor student behaviors when interacting with automatic assessment systems including numerous submissions, trial-and-error, and relying on marking feedback for problem solving. These behaviors negatively affect student learning as well as have significant impact on system resources. This research quantitatively examines how utilizing submission policies such as limiting the number of submissions and applying regression penalties can reduce negative student behaviors. The hypothesis is that both submission policies will have a significant impact on student behavior and reduce both the number of submissions and regressions in student performance. The research questions evaluate the impact on student behavior, determine which submission policy is the most effective, and what submission policy is preferred by students.

Participants. The study involved two course sections in two different semesters consisting of a total of 224 students at the University of British Columbia, a research-intensive university. The students were evaluated using an automated assessment system in a large third year database course.

Study Methods. The two course sections used an automated assessment system for constructing database design diagrams for assignments and exams. The first section had no limits on the number of submissions for both assignments and exams. The second section had limits for the exams but no limits on assignments. On the midterm, participants were randomly assigned to have either a restriction on the total number of submissions or unlimited submissions but with regression penalties if a graded answer was lower than a previous submission. On the final exam, students were given the option of selecting their submission policy. Student academic performance and submission profiles were compared between the course sections and the different submission policies.

Findings. Unrestricted use of automatic grading systems results in high occurrence of undesirable student behavior including trial-and-error guessing and reduced time between submissions without sufficient independent thought. Both submission policies of limiting maximum submissions and utilizing regression penalties significantly reduce these behaviors by up to 85%. Overall, students prefer maximum submission limits, and demonstrate improved behavior and educational outcomes.

Conclusions. Automated assessment systems when used for larger problems related to design and programming have benefits when deployed with submission restrictions (maximum attempts or regression penalty) for both improved student learning behaviors and to reduce the computational costs for the system. This is especially important for summative assessment but reasonable limits for formative assessments are also valuable.