Simplifying the creation and maintenance of automated assessments of programming tasks via Test Specific Language

Creating automated tests for assessing student programs often requires considerable time, work, and knowledge that may exceed the instructor's comfortability threshold. This constitutes an obstacle to adding new tasks to a course, updating the existing ones, or setting up automatic assessment in courses where it is currently missing. This paper presents a system that was developed to meet the goal of making the creation of automated assessment easier for instructors. The system allows the instructor to specify a set of dynamic or static tests to be performed on the student program via specific user interface and it is part of a larger open-source submission and assessment system, Lahendus. The new system produces a tester code that actually assesses the student's submission and provides feedback. The central component of the system is the TSL language for specifying automatic tests. The paper also describes the process of system development and validation, carried out in close collaboration with the intended end-users, i.e., instructors of programming courses.