Assessment Automation of Complex Student Programming Assignments

Grading student programming assignments is not an easy task. This task is even more challenging when talking about complex programming assignments at university graduate level. By complex assignments, we mean assignments where students have to program a complete application from scratch. For example, building a complete web application with a client and server side, whereby the application uses multiple threads that gather data from some external service (like the REST service, IoT sensors, etc.), processes these data and store them in some storage (e.g., a database), implements a custom protocol over a socket or something similar, implements their own REST/SOAP/GraphQL service, then sends or receives JMS/MQTT/WebSocket messages, etc. Such assignments give students an inside view of building real Internet applications. On the other hand, assignments like these take a long time to be tested and graded manually, e.g., up to 1 h per student. To speed up the assessment process, there are different automation possibilities that can check for the correctness of some application parts without endangering the grading quality. In this study, different possibilities of automation are described that have been improved over several years. This process takes advantage of unit testing, bash scripting, and other methods. The main goal of this study is to define an assessment process that can be used to grade complex programming assignments, with concrete examples of what and how to automate. This process involves assignment preparation for automation, plagiarism (i.e., better said similarity) detection, performing an automatic check of the correctness of each programming assignment, conducting an analysis of the obtained data, the awarding of points (grading) for each programming assignment, and other such activities. We also discuss what the downsides of automation are and why it is not possible to completely automate the grading process.