Model solutions and properties for diagnosing student programs in Ask-Elle

Ask-Elle is an interactive tutor that supports the stepwise development of simple functional programs. Using Ask-Elle students receive feedback about whether or not they are on the right track, they can ask for a hint when they are stuck, and get suggestions about how to refactor their program. Our tutor generates this feedback from model solutions and properties that a solution should satisfy. This paper studies the feasibility of using model solutions together with the desired properties of solutions to analyse the work of a student. It describes an experiment in which we analyse almost 3500 log entries from students using Ask-Elle to solve functional programming exercises, to determine how many of these programs are diagnosed correctly based on model solutions and the desired properties of solutions. Ask-Elle manages to correctly diagnose 82.9% of the student programs. A further analysis of the student programs and the diagnoses shows that adding some reasonable model solutions, properties of model solutions, and general program transformations would increase this percentage to 92.9%.