{"title":"Exploring the Effectiveness of Large-Scale Automated Writing Evaluation Implementation on State Test Performance Using Generalised Boosted Modelling","authors":"Yue Huang, Joshua Wilson","doi":"10.1111/jcal.70009","DOIUrl":null,"url":null,"abstract":"<div>\n \n \n <section>\n \n <h3> Background</h3>\n \n <p>Automated writing evaluation (AWE) systems, used as formative assessment tools in writing classrooms, are promising for enhancing instruction and improving student performance. Although meta-analytic evidence supports AWE's effectiveness in various contexts, research on its effectiveness in the U.S. K–12 setting has lagged behind its rapid adoption. Further rigorous studies are needed to investigate the effectiveness of AWE within the U.S. K–12 context.</p>\n </section>\n \n <section>\n \n <h3> Objectives</h3>\n \n <p>This study aims to investigate the usage and effectiveness of the Utah Compose AWE system on students' state test English Language Arts (ELA) performance in its first year of statewide implementation.</p>\n </section>\n \n <section>\n \n <h3> Methods</h3>\n \n <p>The sample included all students from grades 4–11 during the school year 2015 in Utah (<i>N</i> = 337,473). Employing a quasi-experimental design using generalised boosted modelling for propensity score weighting, the analysis focused on estimating the average treatment effects among the treated (ATT) of the AWE system.</p>\n </section>\n \n <section>\n \n <h3> Results and Conclusions</h3>\n \n <p>The results showed that students who utilised AWE more frequently demonstrated improved ELA performance compared to their counterparts with lower or no usage. The effects varied across certain student demographic groups. This study provides strong and systematic evidence to support the hypothesis of causal inferences regarding AWE's effects within a large-scale, naturalistic implementation, offering valuable insights for stakeholders seeking to understand the effectiveness of AWE systems.</p>\n </section>\n </div>","PeriodicalId":48071,"journal":{"name":"Journal of Computer Assisted Learning","volume":"41 2","pages":""},"PeriodicalIF":5.1000,"publicationDate":"2025-02-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://onlinelibrary.wiley.com/doi/epdf/10.1111/jcal.70009","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Journal of Computer Assisted Learning","FirstCategoryId":"95","ListUrlMain":"https://onlinelibrary.wiley.com/doi/10.1111/jcal.70009","RegionNum":2,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"EDUCATION & EDUCATIONAL RESEARCH","Score":null,"Total":0}
Exploring the Effectiveness of Large-Scale Automated Writing Evaluation Implementation on State Test Performance Using Generalised Boosted Modelling
Background
Automated writing evaluation (AWE) systems, used as formative assessment tools in writing classrooms, are promising for enhancing instruction and improving student performance. Although meta-analytic evidence supports AWE's effectiveness in various contexts, research on its effectiveness in the U.S. K–12 setting has lagged behind its rapid adoption. Further rigorous studies are needed to investigate the effectiveness of AWE within the U.S. K–12 context.
Objectives
This study aims to investigate the usage and effectiveness of the Utah Compose AWE system on students' state test English Language Arts (ELA) performance in its first year of statewide implementation.
Methods
The sample included all students from grades 4–11 during the school year 2015 in Utah (N = 337,473). Employing a quasi-experimental design using generalised boosted modelling for propensity score weighting, the analysis focused on estimating the average treatment effects among the treated (ATT) of the AWE system.
Results and Conclusions
The results showed that students who utilised AWE more frequently demonstrated improved ELA performance compared to their counterparts with lower or no usage. The effects varied across certain student demographic groups. This study provides strong and systematic evidence to support the hypothesis of causal inferences regarding AWE's effects within a large-scale, naturalistic implementation, offering valuable insights for stakeholders seeking to understand the effectiveness of AWE systems.
期刊介绍：
The Journal of Computer Assisted Learning is an international peer-reviewed journal which covers the whole range of uses of information and communication technology to support learning and knowledge exchange. It aims to provide a medium for communication among researchers as well as a channel linking researchers, practitioners, and policy makers. JCAL is also a rich source of material for master and PhD students in areas such as educational psychology, the learning sciences, instructional technology, instructional design, collaborative learning, intelligent learning systems, learning analytics, open, distance and networked learning, and educational evaluation and assessment. This is the case for formal (e.g., schools), non-formal (e.g., workplace learning) and informal learning (e.g., museums and libraries) situations and environments. Volumes often include one Special Issue which these provides readers with a broad and in-depth perspective on a specific topic. First published in 1985, JCAL continues to have the aim of making the outcomes of contemporary research and experience accessible. During this period there have been major technological advances offering new opportunities and approaches in the use of a wide range of technologies to support learning and knowledge transfer more generally. There is currently much emphasis on the use of network functionality and the challenges its appropriate uses pose to teachers/tutors working with students locally and at a distance. JCAL welcomes: -Empirical reports, single studies or programmatic series of studies on the use of computers and information technologies in learning and assessment -Critical and original meta-reviews of literature on the use of computers for learning -Empirical studies on the design and development of innovative technology-based systems for learning -Conceptual articles on issues relating to the Aims and Scope
京公网安备 11010802042870号
京ICP备2023020795号-1
分享
求助内容：
应助结果提醒方式：
扫码关注我们
