Are Executable Exams Executable?

Abstract

This lightning talk presents insights from preliminary research on attitudes toward executable exams in an Introduction to Computer Science (CS1) course. In an executable exam, during the exam, students work on a computer in a designated programming environment that enables them to compile their code and run unit tests. Although this exam format is authentic and resembles the way students work during their studies and in the industry, it is not a popular format. We implemented executable exams in the 2021 Winter semester final exam and explored the attitudes toward executable exams of three main groups - students, course staff, and industry representatives. In addition, we collected and analyzed data from high school CS teachers, CS and EE students from other academic institutions and CS lecturers around the world. The attitudes are categorized into three aspects: pedagogical, technical and psychological. All groups expressed concerns regarding technical problems related to computer malfunction and plagiarism. In addition, the course staff and the students expressed concern about the change in the exam format. The groups' attitudes are supported by the pedagogical advantages and disadvantages of executable exams they pointed out.