High-Coverage Hint Generation for Massive Courses: Do Automated Hints Help CS1 Students?

In massive programming courses, automated hint generation offers the promise of zero-cost, zero-latency assistance for students who are struggling to make progress on solving a program. While a more robust hint generation approach based on path construction requires tremendous engineering effort to build, another easier-to-build approach based on program mutations suffers from low coverage. This paper describes a robust hint generation system that extends the coverage of the mutation-based approach using two complementary techniques. A syntax checker detects common syntax misconception errors in individual sub-expressions to guide students to partial solutions that can be evaluated for the semantic correctness. A mutation-based approach is then used to generate hints for almost-correct programs. If the mutation-based approach fails, a case analyzer detects missing program branches to guide students to partial solutions with reasonable structures. After analyzing over 75,000 program submissions and 8,789 hint requests, we found that using all three techniques together could offer hints for any program, no matter how far it was from a correct solution. Furthermore, our analysis shows that hints contributed to students' progress while still encouraging the students to solve problems by themselves.