Educational Measurement-Issues and Practice最新文献

Unlike standardized testing applications of validity, teachers need a simple and efficient way to reflect on the accuracy of the claims based on student performance, then consider whether the uses of those claims are appropriate. A two-phase reasoning process of validation, consisting of a proficiency claim /argument and a use/argument, is presented as a way for teachers to understand and apply the central tenets of validation to their classroom assessments. Since classroom assessment is contextualized with multiple purposes, each teacher is obligated to use validation for their situation. The accuracy of teachers’ conclusions about the proficiency claims, and uses, will depend on their skill in gathering supportive evidence and considering alternative explanations. Examples of the proposed classroom assessment validation process are presented.

The rapid advancement of large language models (LLMs) has enabled the generation of coherent essays, making AI-assisted writing increasingly common in educational and professional settings. Using large-scale empirical data, we examine and benchmark the characteristics and quality of essays generated by popular LLMs and discuss their implications for two key components of writing assessments: automated scoring and academic integrity. Our findings highlight limitations in existing automated scoring systems when applied to essays generated or heavily influenced by AI, and identify areas for improvement, including the development of new features to capture deeper thinking and recalibrating feature weights. Despite growing concerns that the increasing variety of LLMs may undermine the feasibility of detecting AI-generated essays, our results show that detectors trained on essays generated from one model can often identify texts from others with high accuracy, suggesting that effective detection could remain manageable in practice.

Although AI transformer models have demonstrated notable capability in automated scoring, it is difficult to examine how and why these models fall short in scoring some responses. This study investigated how transformer models’ language processing and quantification processes can be leveraged to enhance the accuracy of automated scoring. Automated scoring was applied to five science items. Results indicate that including item descriptions prior to student responses provides additional contextual information to the transformer model, allowing it to generate automated scoring models with improved performance. These automated scoring models achieved scoring accuracy comparable to human raters. However, they struggle to evaluate responses that contain complex scientific terminology and to interpret responses that contain unusual symbols, atypical language errors, or logical inconsistencies. These findings underscore the importance of the efforts from both researchers and teachers in advancing the accuracy, fairness, and effectiveness of automated scoring.

The 2014 Standards for Educational and Psychological Testing describe universal design as an approach that offers promise for improving the fairness of educational assessments. As the field reconsiders questions of fairness in assessments, we propose a new framework that addresses the entire assessment lifecycle: universal design of assessment systems. This framework is rooted in the original Universal Design principles but extends beyond test design and administration to the entire assessment lifecycle, from construct definition to score interpretation and use. Another core tenet within this framework is the integration of psychological theory on universal human needs for autonomy, competence, and relatedness with flexibility based on our contemporary understandings of neurodiversity, culture, and multilingualism. Finally, the framework integrates the original Universal Design principle of tolerance for error, which promotes assessment designs that anticipate unintended actions and mitigate potential harms. After describing how the principles and guidelines might apply in contexts ranging from classroom assessments to statewide assessments and licensure exams, we conclude with practical implications and next steps. We hope future versions of the Standards for Educational and Psychological Testing incorporate this broader, systems-wide approach to universal design.