Effects of Korean Secondary English Teachers’ Perceived Criterion Importance on Scoring Behavior in L2 Writing Assessment

The role of raters in performance-based assessment is crucial, but due to the interactions between raters and rating criteria, rater-induced errors often interfere with the accurate evaluations of examinees’ performances. Based on the hypothesis (Eckes, 2012) that severity bias is combined with higher criterion importance and leniency bias is aligned with lower criterion importance, this study investigates the relationship between perceived criterion importance and scoring behavior by employing hierarchical clustering and a Many-facet Rasch analysis. In a survey, 30 in-service secondary Korean English teachers indicated their perceived criterion importance and rated thirty English essays from Yonsei English Learner’s Corpus (YELC). The results of the survey responses and the measures from the MFRM interaction analysis between participants and the rating criteria were applied respectively to form two rater types: cognitive raters types (CRTs), based on criterion perception and operational rater types (ORTs), based on criterion-related bias. The comparison of these two rater types and the analysis of individual rater’s criterion importance rating together with their criterion-related bias measures showed that the hypothesized link between rater criterion perception and rating behavior only existed in content. The findings of the study will provide valuable information for rater training and Korean English teachers evaluating writing performances.