Comparing computer-based and paper-based rating modes in an English writing test

The study utilized a mixed methods approach to compare the scoring of raters in assessing writing performance across three modes: paper-based, on-screen marking of scanned images, and online word-processed versions. six experienced raters evaluated the performances of 39 test-takers in each mode. The many-facet Rasch model was employed to analyze scoring differences among the rating modes; the semi-structured interview was used to collect raters' perceptions towards performance under the three modes. The findings indicated that the difficulty level was ranked in ascending order of on-screen marking of scanned images, paper-based text, and online word-processed text. Bias analysis revealed interactions between the rater and the mode, as well as between the criterion and the mode. Verbal reports from the raters highlighted four construct-irrelevant factors that could potentially influence scoring under the three modes: convenience for essay overview and word recognition, potential underestimation of word count, and raters' preference for essays in handwriting. Based on the results, recommendations were provided for rater training and essay scoring across different modes.