Educational Assessment Evaluation and Accountability最新文献

In international large-scale assessments, students may not be compelled to answer every test item: a student can decide to skip a seemingly difficult item or may drop out before the end of the test is reached. The way these missing responses are treated will affect the estimation of the item difficulty and student ability, and ultimately affect the country’s score. In the Progress in International Reading Literacy Study (PIRLS), incorrect answer substitution is used. This means that skipped and omitted items are treated as incorrect responses. In the present study, the effect of this approach is investigated. The data of 2006, 2011, and 2016 cycles of PIRLS were analyzed using IRTree models in which a sequential tree structure is estimated to model the full response process. Item difficulty, students’ ability, and country means were estimated and compared with results from a Rasch model using the standard PIRLS approach to missing values. Results showed that the IRTree model was able to disentangle the students’ ability and their propensity to skip items, reducing the correlation between ability and the proportion of skipped items in comparison to the Rasch model. Nevertheless, at the country level, the aggregated scores showed no important differences between models for the pooled sample, but some differences within countries across cycles.

PISA aims to serve as a “global yardstick” for educational success, as measured by student performance. For comparisons to be meaningful across countries or over time, PISA samples must be representative of the population of 15-year-old students in each country. Exclusions and non-response can undermine this representativeness and potentially bias estimates of student performance. Unfortunately, testing the representativeness of PISA samples is typically infeasible due to unknown population parameters. To address this issue, we integrate PISA 2018 data with comprehensive Swedish registry data, which includes all students in Sweden. Utilizing various achievement measures, we find that the Swedish PISA sample significantly overestimates the achievement levels in Sweden. The observed difference equates to standardized effect sizes ranging from d = .19 to .28, corresponding to approximately 25 points on the PISA scale or an additional year of schooling. The paper concludes with a plea for more rigorous quality standards and their strict enforcement.

The current study presents a systematic review of teachers’ data literacy, arising from a synthesis of 83 empirical studies published between 1990 to 2021. Our review identified 95 distinct indicators across five dimensions: (a) knowledge about data, (b) skills in using data, (c) dispositions towards data use, (d) data application for various purposes, and (e) data-related behaviors. Our findings indicate that teachers' data literacy goes beyond addressing the needs of supporting student learning and includes elements such as teacher reflection, collaboration, communication, and participation in professional development. Considering these findings, future policies should acknowledge the significance of teacher dispositions and behaviors in relation to data, recognizing that they are as important as knowledge and skills acquisition. Additionally, prioritizing the provision of system-level support to foster teacher collaboration within in-school professional development programs may prove useful in enhancing teachers’ data literacy.

Assessing student writing assignments and providing effective feedback are a complex pedagogical skill teacher candidates need to master. Scarce literature has closely examined the type of feedback that pre-service high-school teachers spontaneously deliver when assessing student writings, which is the main goal of our study. In a sample of 255 high school teacher candidates, we examined the type of feedback that they provided when assessing two writing assignments that differed in quality. One thousand eight hundred thirty-five comments were analyzed and coded into 11 sub-categories. Results showed that candidates’ feedback not only focused on task performance but also on the writing process. Although candidates provided critical and past-oriented evaluations frequently, they also crafted feedback in a neutral tone and included future-oriented suggestions. Further, feedback varied as a function of candidates’ gender, academic discipline, and students’ quality of writing. Teacher training programs may use this information to design resources to address nuances of feedback provision.

This paper presents findings from a year-long network ethnography into the strategies, networks, and outcomes of More Than a Score (MTAS)—a campaign against standardised testing in UK primary schools. Focusing specifically on the parent-based groups of the organisation, we use theorisations of symbolic capital to challenge traditional understandings of how capital can be leveraged for group advancement. We argue that MTAS frames itself as a grassroots organisation, using this image to promote its agenda amongst possible allies. Parent groups serve a critical role in accentuating the ‘grassroots’ image, as they bring a level of credibility to this claim. At the same time, the individuals who run these groups also bring technical skills, professional experiences, and connections that provide logistical and expert capital to the range of MTAS’s strategies and agendas. In doing so, their political and social capital as ‘parents’ provide a sort of legitimising capital to MTAS.

Scores from observational measures of teaching have recently been put to many uses within school systems, including communicating a standard of practice and providing teacher feedback, identifying teachers for professional development, monitoring system equity, and making employment decisions. In each of these uses, observation scores are interpreted as representing some aspect of the enacted instruction or teachers’ capacity to enact instruction, as seen through the observation systems lens for understanding teaching quality. The quality of these interpretations, or the extent to which observation scores are composed of a signal that accurately reflects the interpretation, has important implications for the overall validity of uses of observation systems. Starting from an explicit conceptualization of instruction, this paper combines generalizability theory and hierarchical linear modelling approaches to decompose observation scores to explore the extent to which scores from observation systems are composed of signal, error, and bias across four different uses (i.e., teacher feedback, professional development, monitoring system equity, and employment decisions) of scores. We show that the quality of observation scores may depend more on what scores are interpreted as representing (i.e., the proposed use) than on the specific observation rubric being used. Further, we show that rater errors and biases are a major threat to any attempt to interpret observation scores as capturing the observation system’s understanding of teaching quality. We discuss implications for using scores from observation systems.

Many schools aim to implement Assessment for Learning (AfL) to stimulate students to take more ownership of their learning and develop self-regulatory skills. This survey-based study is among the few in the field showing how students experience the extent of implementation of AfL, here in English language and mathematics classes in 12 Dutch secondary schools. Analysis showed no differences between the subjects. Students (N = 685) experienced activities to clarify the learning intentions and success criteria and eliciting evidence on their learning progress regularly (between 50–74% of the lessons). Students hardly ever experienced activities aimed at peer- and self-assessment (less than 25% of the lessons). However, cluster analysis revealed three distinct clusters related to the extent of AfL strategy use students experienced. Overall, we can conclude that AfL is not yet fully integrated into teaching practices. Teachers’ skills, knowledge and attitudes required to increase student engagement in AfL practices and strengthen students’ self-regulated learning need more attention in future teacher professional development trajectories.

The present study investigates whether alternatively and traditionally certified teachers differ in their teaching quality. We conducted doubly latent multilevel analysis using a large-scale dataset with student ratings on the quality of instruction provided by 1550 traditionally and 135 alternatively certified secondary school mathematics teachers in Germany. Findings show no evidence of differences in teaching quality between alternatively and traditionally certified teachers. Independent of the type of certification, novice teachers scored lower on classroom management than experienced teachers. Findings suggest that beginning teachers—whether alternatively or traditionally certified—need to be better prepared for classroom management before starting to teach.

Abstract

Accountability policies through standardized testing are widespread in diverse educational systems. Based on an ethnographic research study, we sought to understand how a learning assessment policy, used for over three decades in primary and lower secondary education, is lived and interpreted in daily school life in Chile. This article analyzes how learning assessment policies are re-contextualized (translated and interpreted) in three elementary and high schools located in poor neighborhoods with different performance categorization according to students’ test results. We conducted interviews and observations with teachers and school leadership team members. Results show that school actors experience the effects of testing both as a menace and as a pressure despite their different school categorization performance. They consider the tests a decontextualized and unjust measurement and therefore respond with similar strategies that seek to avoid these negative consequences. This study contributes to understanding the effects of tight accountability systems that attempt to place different levels of pressure through categorization.

Based on the theoretical approaches of social capital and institutional trust, this paper seeks to identify contextual factors and conditions behind teacher behaviours which aim to alter the results of standardised tests in the Italian low-stakes accountability system. Numerous studies report significant factors associated with student cheating, but research into the factors of teacher-led opportunistic actions is scarce. Logistic regression models with fixed-effects at classroom level, with interaction terms, were carried out to identify factors increasing the likelihood of teacher misbehaviour. Models included approximately 79,100 primary, lower and upper secondary classrooms. Indicators of teacher cheating were estimated through algorithms based on suspicious answer strings from standardised tests. The results suggest that teacher cheating may be understood as a form of support for the most vulnerable students, since it is, to a greater extent, found helping low-income students, grade-retained students, as well as students in socially homogenous school settings. The findings also reveal that teacher cheating is consistently related to collectively share non-civic-minded behaviours and practices undertaken by teachers, which do not match legal requirements, such as within-school social segregation and exclusion of students from tests. Heterogeneous effects show that, even in classrooms with external controllers, the lower the civic capital in a school, the more misbehaviour are found. Relevant implications for research, social theory and policy are discussed.