Rater Connections and the Detection of Bias in Performance Assessment

Abstract

ABSTRACT In many performance assessments, one or two raters from the complete rater pool scores each performance, resulting in a sparse rating design, where there are limited observations of each rater relative to the complete sample of students. Although sparse rating designs can be constructed to facilitate estimation of student achievement, the relatively limited observations of each rater can pose challenges for identifying raters who may exhibit scoring idiosyncrasies specific to individual or subgroups of examinees, such as differential rater functioning (DRF; i.e., rater bias). In particular, when raters who exhibit DRF are directly connected to other raters who exhibit the same type of DRF, there is limited information with which to detect this effect. On the other hand, if raters who exhibit DRF are connected to raters who do not exhibit DRF, this effect may be more readily detected. In this study, a simulation is used to systematically examine the degree to which the nature of connections among raters who exhibit common DRF patterns in sparse rating designs impacts the sensitivity of DRF indices. The use of additional “monitoring ratings” and variable rater assignment to student performances are considered as strategies to improve DRF detection in sparse designs. The results indicate that the nature of connections among DRF raters has a substantial impact on the sensitivity of DRF indices, and that monitoring ratings and variable rater assignment to student performances can improve DRF detection.