Studies in Educational Evaluation最新文献

This paper addresses the theoretical and practical question of how to specify criteria used in educational evaluations. People and groups involved in and affected by an educational initiative typically bring different values to bear on the question of what a quality or successful initiative means. This poses a challenge of balancing and prioritizing between differing values when specifying criteria. To address this challenge, we present a framework and process that involves systematic consideration of multiple sources of criteria (i.e., who and where) and domains (e.g., design, outcomes) followed by explicitly defining criteria within an evaluation. We illustrate our use of this framework in three evaluations: a high school mathematics teacher program, K-12 principal professional development initiatives, and a graduate-level online healthcare administration program. Together, the framework and illustrations provide guidance and highlight future directions for strengthening explicit criteria specification in educational evaluation.

While recent decades have witnessed the proliferation of studies on peer feedback in L2 writing, little is known about how L2 learners engage affectively, behaviorally, and cognitively with this practice. To narrow this gap, the present study employing a mixed-methods approach examined L2 learners’ engagement with peer feedback in the Chinese secondary school context. Data were collected from a variety of sources over 12 weeks, including questionnaires, semi-structured interviews, writing journals, and learners’ first and revised writing samples. The results showed that the teacher’s scaffolding with systematic instruction comprising pre-feedback sessions, multiple feedback practices, and post-feedback reinforcement helped the participants engage with peer feedback proactively in affect, behavior, and cognition. Furthermore, the participants’ perception of their improvement in the three dimensions was also evident over the semester. Overall, this study demonstrates the role of teachers in student engagement and advances our understanding of L2 learner engagement with peer feedback. Additionally, it offers important pedagogical implications for fostering and promoting L2 learners’ engagement.

The aim of this study was to examine how achievement varied within and between schools at different grade levels, and long-term trends in variation within and across multiple countries. We used science achievement data from five cycles of Trends in International Mathematics and Science Study (TIMSS) from 2003 to 2019 involving 10 countries from Asia, Europe, and the United States. Employing exploratory data mining methods of variance decomposition, correlation analysis, and Gaussian mixture modeling of data distributions, we found the following: First, between-school variances generally remained consistent across two decades, suggesting that inequality between schools has not increased over time. Second, between-school variances were relatively small for elementary grade level but increased at secondary grade level, though marginally even for countries with early tracking. Third, higher-achieving schools tended to have more equal student achievement levels than lower-achieving schools, lending within-country support for the “virtuous” efficiency-equality trade-off. We further found that reduced equality within lower-achieving schools was associated with bimodality in achievement distribution. Overall, there is little evidence of inequality across schools changing over time. However, there may be evidence of increased inequalities associated with student subpopulations, particularly within lower-achieving schools, with implications on classroom instruction and school cohesion.

Background

Classic assessment methods present negative emotional alterations for students, such as stress, anxiety, fear and nervousness; these could be solved by applying the Escape Room, which is a promising tool where students experience positive emotions that are critical to learning.

Aim

To explore the experiences and perceptions of physiotherapy students regarding the use of an Escape Room game-based model for their assessment in contrast to conventional approaches.

Methods

A phenomenological-hermeneutical study was conducted. The assessment took place in two different modalities and on different days: 1) a traditional assessment method and 2) an assessment method incorporating a game-based model. All students participated in the two assessment processes. Fifty-six physiotherapy students took part in this study.

Results

The detailed analysis of the results allowed us to classify them into two main themes: 1) Strengths of the Escape Room as part of an assessment approach and, 2) Weaknesses of the Escape Room as part of an assessment approach. In turn, from these main themes emerged the sub-themes and their different units of meaning.

Conclusions

The results suggest that new teaching and assessment methodologies incorporating innovative models such as the Escape Room are effective in evaluating the skills and performance of physiotherapy students, which can be used to complement the traditional assessment methods.

Based on data from a large-scale assessment of school quality in China, this study examined the differences in the impact of learning and teaching strategies on the students’ academic achievements between elementary and middle schools. Two-level hierarchical linear models and coarsened exact matching revealed three major findings: (a) Cognitive, meta-cognitive, and inquiry-based learning strategies were significant predictors of students’ academic achievements, and the effect sizes of these strategies were larger in middle schools; (b) Individualized instruction, participatory instruction, and guided inquiry teaching strategies were significant predictors of students’ achievements, and the effect sizes of these strategies were larger in elementary schools; (c) Overall teaching strategies significantly improved learning strategies, and the effects were greater in elementary schools. These findings call for more research on learning/teaching strategies to improve students' academic performance.

Much attention has been paid to the relationship between parent-child relationships and adolescent academic performance, especially the mediating role of subjective educational expectations of parents and children. However, most of the research is a single study on parents' or children's educational expectations, and there is a lack of overall research on the independent components and combined effects of the subjective educational expectations system. Based on a large sample of 8867 middle school students, this study showed that parental expectation and self-expectation not only had independent indirect effects on parent-child relationship and academic performance, but also had a series of mediating effects together. The parental expectation was the strongest indirect influencing factor. These findings contribute to a comprehensive understanding of the deep mediating mechanism of educational expectations between parent-child relationship and academic performance in adolescents, and provide important new ideas for educational design practices in home-school co-education.

Differential item functioning (DIF) analysis is essential to ensuring the equity of measurement for different subgroups at the item level and is an integral part of validity. However, existing DIF research often overlooks within-group heterogeneity, commonly assuming that test takers from different subgroups comprise a homogeneous population. This study investigated DIF across gender, academic background, and their interaction in listening comprehension assessment using Rasch measurement. It found that ignoring within-group heterogeneity would lead to the under-detection of DIF, likely due to the cancellation of DIF at broader group levels. In addition, the study is the first to investigate DIF in a linked test, a scenario more prevalent in practical testing. The findings of the study highlight the importance of accounting for within-group heterogeneity in test fairness investigations in language assessment research and point to the potential effect of test linking and equating on DIF analysis and interpretation.

While the body of research on student feedback literacy in the field of education is growing, the association of such literacy with feedback engagement in L1 writing education remains relatively unexplored. Moreover, the potential relationship of a growth mindset, which serves as a motivational factor, with student feedback literacy has yet to be thoroughly investigated. Considering these gaps, the present study aimed to explore the relationships between a growth mindset, student writing feedback literacy, and feedback engagement. A sample of 236 sophomore English major students from a Chinese university was recruited. Employing path analysis, the study revealed that having a growth mindset was positively associated with all five features of student writing feedback literacy: appreciating feedback, acknowledging different feedback sources, making judgments, managing affect, and taking action. Moreover, the study revealed a nuanced relationship between students’ writing feedback literacy and their feedback engagement, with different features of writing feedback literacy having differential associations with distinct dimensions of feedback engagement. The study contributes to the literature in three ways. First, it expands our knowledge of the relationship between student feedback literacy and feedback engagement by delving deeper into the one-to-one association between different features of writing feedback literacy and various dimensions of feedback engagement. Second, it advances our understanding of how a growth mindset is related to shaping students’ writing feedback literacy. Finally, it provides evidence supporting the construct validity of the student writing feedback literacy scale in L1 writing learning.

Second language (L2) writing teachers’ personal beliefs can shape their feedback-giving practices. As a burgeoning area in L2 education, the mindsets framework offers us a novel theoretical perspective to analyze the impact of personal belief on feedback-giving practices. With 288 university English writing teachers from 28 provinces in China, this study examined their growth and fixed mindsets, as well as the five types of feedback-giving practices: scoring feedback, written corrective feedback, process-oriented feedback, expressive feedback, and peer and self-feedback. Paired sample t-test reported that teachers held significantly higher levels of growth than fixed mindsets. Repeated measures analysis of variance results indicated that expressive feedback was the most frequently used feedback approach, whereas written corrective feedback the least used approach. Structural equation modeling results found that growth mindsets were positively associated with all five types of feedback approaches, while fixed mindsets were only positively related to written corrective and process-oriented feedback. We interpreted the underlying mechanisms behind the varied associations between growth versus fixed mindsets with the five types of feedback-giving practices, and proposed suggestions for L2 writing teacher education.

Educational research has focused increasingly on comparing student civic outcomes across various educational systems. However, the study of non-cognitive, attitudinal measures has assumed a subsidiary role to the study of cognitive measures, despite its potential to shape policy and practice. This article investigates the validity and cross-country measurement equivalence or invariance (ME/I) of political trust measures among adolescents. Drawing on student data from ICCS 2016 covering 15 European educational systems (N = 52,788), the standard test for exact ME/I (i.e., Multi-Group Confirmatory Factor Analysis) is juxtaposed with a novel method for approximate ME/I (i.e., Alignment Optimization). The results underscore a two-dimensional structure of political trust, separating trust in order institutions from trust in representative institutions. While MGCFA results suggest configural invariance, rendering correlational analyses and country mean comparisons meaningless, Alignment Optimization yields valid latent means with an acceptable degree of non-invariance (i.e., 16%). Implications for comparative research on political trust among adolescents are discussed.