Using Adaptive Comparative Judgement to Assess Student Work in an MBA Course

Abstract

A common instructional problem with large classes is the assessment of non-standardized student work such as short-answer questions, papers, and projects. The large grading load associated with assessing open-ended work often leads to a greater reliance on multiple-choice questions. While multiple-choice questions provide beneficial information about student performance, they may fail to capture other elements of student performance in regards to communication, writing, and generation of ideas. One potential solution to this problem is the concept of adaptive comparative judgment (ACJ). ACJ is based on the simple premise that while peers may not have the ability to place an objective grade on a paper, they can competently compare two different papers and choose which is superior. With a large enough number of student “judgements” on a body of peer work, the collective results from the comparison process can produce rankings that are on par with how the instructor would rank these papers. This session will highlight the instructor’s use of ACJ with an MBA problem-based class. ACJ was used over a number of consecutive assignments, and it produced a high correlation with the instructor’s rankings. Students also expressed satisfaction with the system and may have benefited from seeing other (anonymous) student work. ACJ represents a promising and fairly easy-to-use approach for grading open-ended work in large classes.