Impact of reflection in auto-graders: an empirical study of novice coders

ABSTRACTBackground and Context Auto-graders are praised by novice students learning to program, as they provide them with automatic feedback about their problem-solving process. However, some students often make random changes when they have errors in their code, without engaging in deliberate thinking about the cause of the error.Objective To investigate whether requiring students using an auto-grading system to reflect on the errors in their code would reduce trial and error behavior often seen in such systems.Method The paper analyzes the impact of reflection per student and per problem using paired t-tests.Findings Students took fewer steps to solve the problem in reflective sessions than in Usual Debugging Sessions (4.33 vs 8.04) and they made half as many syntax errors, logic errors, and runtime errors. However, they took more time between runs.Implications This paper provides evidence that requiring reflection in autograding systems can improve student debugging skills.KEYWORDS: Reflective debuggingintroductory programming classesonline IDEactionable learning analyticsauto gradersnovice programming Disclosure statementNo potential conflict of interest was reported by the author(s).Ethical approvalThis study was done in accordance with the university IRB with reference #15041.Data availability statementThe datasets generated during and/or analyzed during the current study https://doi.org/10.6084/m9.figshare.24188010