Interrater agreement in genre analysis: A methodological review and a comparison of three measures

This study examines the coding procedures, practices, and measures of interrater agreement (IRA) for conducting rhetorical move analysis. We first systematically reviewed 65 English for Academic Purposes (EAP) genre studies, focusing primarily on their IRA rationales and practices. Based on the review outcome, we explored three IRA measures: percent agreement, Kappa coefficients (κ), and Krippendorff's multi-valued alpha (mv-α) in terms of their suitability for measuring IRA in rhetorical move analysis under different scenarios such as different sample sizes, number of coders, and multivalued coding. Specifically, the three researchers annotated 35 texts from the Corpus of Social Sciences Research Article Introductions (COSSRAI) (Lu et al., 2020) and subsequently compared the potential of the three IRA measures. Drawing on findings of the systematic review and the corpus study, we recommended strategies for ensuring methodological rigors and transparency in annotation and IRA practices for EAP genre analysis.