Evaluation of Programming Skills via Peer Assessment and IRT Estimation Techniques

Abstract

In whatever study program where Computer Science is taught (as a support subject matter, or as the main one) the analysis of students’ programming skills is a complex and crucial endeavour. Peer assessment (PA) can be used to expose students (peers) to a very effective educational methodology, to spur competence, and to evaluate skills in a wide range of subject matters, including Computer Science and programming. An important feature is in that data from PA sessions can be used to model the students, and support the inference of automated grading. In this paper we analyse the data coming from experiments where several PA sessions were conducted, with students having to produce programs, and evaluate their peers’ programs. The main aim is to see how methods of the Item Response Theory (IRT) can be applied in the PA framework, to model the students effectively. The results seem encouraging, allowing to foresee the enrichment of more traditional automated grading techniques by the IRT methods.