Online formative assessment using automated essay scoring technology in China and U.S.— Two case studies

Abstract

Automated essay scoring (AES) has benefits such as time saving for teachers, immediate diagnostic feedback and reliable score reporting. This paper reports two case studies of using online automated essay scoring technology as formative assessment tool in the Chinese and the U.S. classrooms. In the first case study, we report how Writing Roadmap 2.0 (WRM 2.0) [6], an online formative assessment tool, is integrated in the West Virginia classrooms, U.S.A. In the second case study, we summarize a pilot study on Writing Roadmap 2.0 use in the English writing classrooms in Beijing. We present two-year empirical data from the state of West Virginia online writing assessments to exam the effects of classroom formative assessment using Writing Roadmap 2.0 AES program on the state summative writing assessment scores. Consistent and statistically significant positive mean score differences on the state summative writing test were found for students practiced using WRM 2.0 software compared to students who did not practice. The largest score gain was found for low performing students who practiced 4 or more times compared to no practice students from the same performance level. In the case study in Beijing China Foreign Affairs University (CFAU), we found that the AI-scoring writing program alone is not sufficient for improving student writing. EFL English teachers need professional development to integrate online formative assessment in teaching. Teachers and students experience role changing when AI essay scoring software is used in the classroom. Understanding the online writing score feedback and transforming the immediate score data information to actions is the key to success.