What do exam results really measure?

Abstract

Students are evaluated using examinations, but how do we evaluate whether the examination is correctly measuring the students’ knowledge or skill? This paper presents a methodology which we have used in an experiment: students’ exam results were analysed to reveal which different cognitive skills were used in answering different questions. The analysis revealed that students were approaching several questions in ways that the instructor had not anticipated. Sometimes questions the instructor considered straightforward actually tested students’ conceptual understanding; on other questions which were intended to require problem-solving, many students never identified the concepts involved, so that grades measured primarily the ability to avoid distraction. In other cases, we demonstrated that questions did assess deep understanding of the fundamental concepts, rather than rote-learning or simple patternmatching.