Studies in Educational Evaluation最新文献

This paper evaluates the potential of generative artificial intelligence (AI) in higher education, specifically its capacity to replace or assist human teachers. By reviewing relevant literature and analysing survey data from students and teachers, this mixed-methods study provides a comprehensive perspective on the future role of educators in the face of advancing generative AI technologies. An online survey was conducted to explore the perceptions of 399 university students and 184 teachers across different disciplines in eight higher education institutions in Hong Kong concerning the use of generative AI technologies. Findings suggest that although some believed generative AI may eventually replace teachers, the majority of participants argued that human teachers possess unique qualities, including critical thinking and emotions, which make them irreplaceable. Similarly, findings also emphasized the importance of social-emotional competencies developed through human interactions, something which generative AI technologies cannot currently replicate. Crucially, this study further found that students value and respect their human teachers, even as generative AI becomes more prevalent. As such, the authors propose that teachers can seek to effectively integrate generative AI to enhance teaching and learning without viewing it as their replacement. To do so, they must understand how generative AI can work well with teachers and students, avoid potential pitfalls, develop AI literacy, and address practical issues including ethics and privacy. Recommendations are offered on how universities, teachers, and students can adopt generative AI technologies in an approach that balances the strengths of human educators with generative AI technologies. As the future of education lies in the synergy between human teachers and generative AI, teachers, students, and universities should all understand and refine their unique qualities in order to effectively navigate the integration of generative AI, ensuring well-rounded and impactful learning experiences.

This study presents the results of a qualitative evaluation of how pre-service teachers incorporate games into their classroom practices during internships. Two Game-Based Learning (GBL) methods, namely the PCaRD model and the GameChangers approach, were introduced to assist pre-service teachers in delivering game-based learning activities effectively within disciplines such as English and Sports, as part of their university course requirements. Data was collected from 48 pre-service teachers, highlighting the perceived benefits of both methods in teaching with games and emphasizing their complementarity. Essential competencies identified included prior gaming experience, training in game creation, digital skills, and understanding of student behavior and reactions. Main challenges encountered included time constraints, classroom equipment deficiencies, class management issues, pupil behavior, and aligning games with the curriculum.

International large-scale assessments such as TIMSS administer achievement tests that are based on an analysis of national curricula to compare student achievement across countries. The organizations that coordinate these studies use Rasch or more generalized item response theory (IRT) models in which all test items are assumed to measure a single latent ability. The test responses are then used to estimate this ability, and the ability scores are used to compare countries.

A central but yet-to-be-contested assumption of this approach is that the achievement tests measure an unobserved unidimensional continuous variable that is comparable across countries. One threat to this assumption is the fact that countries and even regions or school tracks within countries have different curricula. When seeking to fairly compare countries, it seems legitimate to account for the fact that applicable curricula differ and that some students may not have been taught the full test content yet. When seeking to fairly compare countries, it seems imperative to account for the fact that national curricula differ and that some countries may not have taught the full test content yet. Nevertheless, existing IRT-based rankings ignore such differences.

The present study proposes a direct method to deal with differing curricula and create a fair ranking of educational quality between countries. The new method compares countries solely on test content that has already been taught; it uses information on whether students have mastered skills taught in class or not and does not consider contents that have not been taught yet. Mastery is assessed via the deterministic-input, noisy, “and” gate (DINA) model, an interpretable and tractable cognitive diagnostic model. To illustrate the new method, we use data from TIMSS 1995 and compare it to the IRT-based scores published in the international study report. We find a mismatch between the TIMSS test contents and national curricula in all countries. At the same time, we observe a high correlation between the scores based on the new method and the conventional IRT scores. This finding underscores the robustness of the performance measures reported in TIMSS despite existing differences across national curricula.

This study focuses on the role of data coaches in data-based decision-making (DBDM) in education. Interviews and focus groups with academics and practitioners were conducted to understand the needs and challenges regarding data coaching. The study highlights the collaborative nature of data coaching and suggests the involvement of multiple coaches, both internal and external to the school. Distinctions between primary and secondary education contexts are revealed. The practitioners encountered challenges in articulating the specific practices, knowledge, and skills required for a data coach. Despite these challenges, the study provides valuable insights for future research and contributes to a comprehensive understanding of data coaching in education. This study increases the understanding of the various requirements and preferences related to the role of data coaches in promoting effective data-based decision making practices, as data coaches and school teams collaborate closely.

Insights into student behavioral patterns yield benefits for both educators and learners. Nevertheless, only a few studies have investigated the behavioral patterns of high-scoring (HS) and low-scoring (LS) students to facilitate predictions during peer assessment (PA). Therefore, we performed learning analytics to explore the behavioral patterns of HS and LS students in 52 university students from affective, cognitive, and metacognitive perspectives as these students engage in online PAs. The results indicated that on the affective dimension, HS students tended to exhibit negative affection while LS students tended to display positive affection. On the cognitive dimension, HS students demonstrated more intricate transformations compared to the LS students. Regarding the metacognitive dimension, LS students more likely reflected upon and accepted the reviews provided by others than the HS. Additionally, the findings also revealed that HS students were supported by focused social networks and overall stable activity engagement, whereas the LS students were less engaged at the beginning of the activity but had evolving social networks and were more engaged in the later stages of the activity. These findings offer valuable insights for educators in effectively predicting student performance based on behavioral patterns while providing new perspectives to support instructional decision-making processes.

This longitudinal case study investigates three EFL learners’ engagement changes with peer feedback giving from affective, behavioral and cognitive perspectives over an academic year. Data were collected from multiple sources, including learners’ texts, feedback messages, open-ended questionnaire, stimulated recalls, and semi-structured interviews. The findings show that the three dimensions of learner engagement with peer feedback giving interacted with each other dynamically during the academic year. While learners’ positive affective engagement promoted their cognitive and behavioral engagement, their negative feelings did not necessarily restrain the enhancement of the other two aspects. In turn, learners’ cognitive and behavioral engagement could either promote or hinder their affective engagement. This study advances our understanding of the multifaceted and dynamic nature of learner engagement with peer feedback giving. It provides pedagogical implications for writing educators to develop teaching strategies to promote learner engagement and maximize learning opportunities in the feedback giving process.

Mixed teaching strategies have caused a fierce debate about teaching quality in China's higher education. Based on theories of constructivism and self-regulation, we examine the impact of mixed teaching strategies on learners' ambidextrous innovation capacities. Utilizing a questionnaire survey of 412 undergraduate and graduate students from 36 Chinese higher education institutions, this research demonstrates that combining metacognitive strategies with motivational and collaborative approaches enhances exploratory innovation ability, while the pairing of metacognitive and collaborative strategies promotes exploitative innovation. Furthermore, the study identifies significant differences in the effectiveness of mixed teaching strategies based on students' self-efficacy and teaching acceptance levels. These findings highlight the importance of tailoring mixed teaching strategies to individual characteristics for optimal learning outcomes and fostering students' innovative potential. This research contributes to the discourse on effective teaching practices in Chinese higher education by proposing a mixed teaching strategies matrix informed by self-efficacy and teaching acceptance, offering valuable guidance for educators seeking to cultivate students' innovative abilities.

The capability of scales to detect change, i.e. sensitivity to change, is important in longitudinal studies. We present a quantitative study comparing the effects of alignment to course content and item granularity (generic vs. concrete wording) on the capability of different scales to capture changes in ICT self-efficacy. In an evaluation of a course on teaching with ICT, we used ICT self-report questionnaires for pre-service teachers at the beginning and end of a one-semester in a treatment and a control group (N_intervention = 278 vs. N_control = 395). After propensity score matching on the imputed data, we applied descriptive, frequentist, and Bayesian methods to examine differential scale behavior. Results show more pronounced longitudinal changes for scales on instructional ICT self-efficacy, especially for scales formulated for concrete tasks. Our results suggest that better alignment and specific formulation enhance sensitivity to change in self-reports, and relate to scale construction and selection in educational evaluations.

Recent years have seen a surge in higher education educators employing technology to support students' holistic competency development. A rigorous evaluation is essential to ensure these technological interventions achieve their intended outcomes. This systematic review of 37 studies from January 2008 to March 2024 evaluates how technology assists in enhancing teamwork competencies in higher education. Using meta-ethnography, it synthesizes qualitative student feedback to assess the effectiveness and challenges of technology use. The findings reveal that technology integration helps students realize the importance of teamwork, understand team dynamics, develop practical skills, and foster team spirit. Additionally, technology's contribution to teamwork training includes boosting student engagement, facilitating the execution of team tasks, and advancing student self-reflection. However, technological interventions also present challenges, including increased training complexity, deviation from learning objectives, and negative impacts on motivation. These findings are crucial for educators aiming to develop effective technology-enhanced training programs to improve teamwork competencies.

We examined how teacher support and disciplinary climate may have additional benefits for students from low (or high) socioeconomic status (SES) families in high and low globally competitive economies. Multilevel regression analyses on 184102 students from 6397 schools from diversified globally competitive economies in the Programme for International Student Assessment (PISA) showed that: (a) students in high SES families benefited more in reading in higher than lower globally competitive economies; (b) greater teacher support improved reading in high than low globally competitive economies; (c) teacher support would alleviate the negative impact of low SES on reading; and (d) a positive disciplinary climate enhances reading performance in both high and low globally competitive economies, with SES effects being more pronounced in schools with a positive disciplinary climate, especially in low globally competitive economies. Practical implications, limitations, and future research directions are discussed.