Large-Scale Assessments in Education最新文献

As the use of process data in large-scale educational assessments is becoming more common, it is clear that data on examinees’ test-taking behaviors can illuminate their performance, and can have crucial ramifications concerning assessments’ validity. A thorough review of the literature in the field may inform researchers and practitioners of common findings as well as existing gaps. This literature review used topic modeling to identify themes in 221 empirical studies using process data in large-scale assessments. We identified six recurring topics: response time models, response time-general, aberrant test-taking behavior, action sequences, complex problem-solving, and digital writing. We also discuss the prominent theories used by studies in each category. Based on these findings, we suggest directions for future research applying process data from large-scale assessments.

Abstract

The benefits of incorporating process information in a large-scale assessment with the complex micro-level evidence from the examinees (i.e., process log data) are well documented in the research across large-scale assessments and learning analytics. This study introduces a deep-learning-based approach to predictive modeling of the examinee’s performance in sequential, interactive problem-solving tasks from a large-scale assessment of adults' educational competencies. The current methods disambiguate problem-solving behaviors using network analysis to inform the examinee's performance in a series of problem-solving tasks. The unique contribution of this framework lies in the introduction of an “effort-aware” system. The system considers the information regarding the examinee’s task-engagement level to accurately predict their task performance. The study demonstrates the potential to introduce a high-performing deep learning model to learning analytics and examinee performance modeling in a large-scale problem-solving task environment collected from the OECD Programme for the International Assessment of Adult Competencies (PIAAC 2012) test in multiple countries, including the United States, South Korea, and the United Kingdom. Our findings indicated a close relationship between the examinee's engagement level and their problem-solving skills as well as the importance of modeling them together to have a better measure of students’ problem-solving performance.

Meta-analyses and international large-scale assessments (ILSA) are key sources for informing educational policy, research, and practice. While many critical research questions could be addressed by drawing evidence from both of these sources, meta-analysts seldom integrate ILSAs, and current integration practices lack methodological guidance. The aim of this methodological review is therefore to synthesize and illustrate the principles and practices of including ILSA data in meta-analyses. Specifically, we (a) review four ILSA data inclusion approaches (analytic steps, potential, challenges); (b) examine whether and how existing meta-analyses included ILSA data; and (c) provide a hands-on illustrative example of how to implement the four approaches. Seeing the need for meta-analyses on educational inequalities, we situated the review and illustration in the context of gender differences and socioeconomic gaps in student achievement. Ultimately, we outline the steps meta-analysts could take to utilize the potential and address the challenges of ILSA data for meta-analyses in education.

Test-taking behavior is a potential source of construct irrelevant variance for test scores in international large-scale assessments where test-taking effort, motivation, and behaviors in general tend to be confounded with test scores. In an attempt to disentangle this relationship and gain further insight into examinees’ test-taking processes, researchers can now utilize process and timing data to obtain a more comprehensive view of test-taking behaviors, such as test-taking effort. The purpose of this study is to propose and evaluate two novel response-based, standardized indicators of test-taking behaviors that utilize a combination of examinee response and process (timing) data to better understand and describe test-taking effort in ILSAs. These indices were empirically estimated with USA data from two booklets from e-TIMSS 2019 in mathematics for grade 4. In addition, their predictive validity was examined with respect to achievement estimates. Their network of associations with other relevant variables such as motivation, interest in the subject, as well as across subjects were also examined to test their intra-individual stability in e-TIMSS.

This paper aims to showcase the value of implementing a Bayesian framework to analyze and report results from international large-scale assessments and provide guidance to users who want to analyse ILSA data using this approach. The motivation for this paper stems from the recognition that Bayesian statistical inference is fast becoming a popular methodological framework for the analysis of educational data generally, and large-scale assessments more specifically. The paper argues that Bayesian statistical methods can provide a more nuanced analysis of results of policy relevance compared to standard frequentist approaches commonly found in large-scale assessment reports. The data utilized for this paper comes from the Teaching and Learning International Survey (TALIS). The paper provides steps in implementing a Bayesian analysis and proposes a workflow that can be applied not only to TALIS but to large-scale assessments in general. The paper closes with a discussion of other Bayesian approaches to international large-scale assessment data, in particularly for predictive modeling.

In this study we use data from the Programme for International Student Assessment (PISA) to investigate the effect of teachers’ motivation on students’ scientific literacy and motivation in Colombia. These relationships are explored using a multilevel modeling framework and through the lens of Self-Determination Theory. Although difficulties in achieving education quality in developing and emerging economies are commonly attributed to teacher motivation issues, and important policy measures are implemented based on this assumption, this topic remains largely empirically unexplored. The purpose of the study is to contribute to fill this gap and provide empirically based insights for a broader and more informed dialogue regarding the effect of motivation in the development of scientific literacy, and to the design and implementation of evidence-based policies, instructional practices, and interventions. In this analysis, we did not find a significant relationship between teacher motivation and either students’ scientific literacy or motivation. However, students’ interest in science and sense of self-efficacy were significantly associated with their own achievement. The results also show that teacher-directed instruction is the strongest predictor of scientific literacy as opposed to inquiry-based teaching. However, inquiry-based teaching was found to be a positive predictor of increased students’ motivation.

Background

The global outbreak of the novel COVID-19 virus presented a significant threat to students’ well-being across the globe. In this paper, we construct a measure of student psychological distress related to COVID-19 disruption. We then examine the variation in students’ psychological distress as a function of student demographic characteristics, home, school and student factors and compare the results across countries.

Methods

We use item response theory to construct a comparable scale for students’ psychological distress across participating countries. Furthermore, we employ linear regression to explore the association of student characteristics and other student and school factors.

Results

An internationally comparable scale for students’ psychological distress was constructed using the model assuming equal item parameters across countries. This enables us to compare the levels of students’ psychological distress and its relationships with the construct across countries. The most important factors contributing to students’ psychological distress were school support, school belonging, disrupted sleep, difficulties in learning after the disruption and preparedness for future disruptions. In some countries, we find suggestive evidence that boys exhibited lower psychological distress than girls. We do not find any meaningful relationship between home resources and the students’ psychological distress scale.

Conclusions

Students across participating countries expressed negative feelings about schooling and events happening during the disruption and their effects on their future. We find indication that some school and student factors had a significant relationship with students’ psychological distress in many countries. This was especially the case in countries where remote learning took place during the disruption. In addition, differences across countries are found. The key finding is that high psychological distress is present in all countries studied around the world. However, it is important to note that the factors contributing to this distress are not the same everywhere. Therefore potential interventions must consider country specific factors.

A general validity and survey quality concern with student questionnaires under low-stakes assessment conditions is that some responders will not genuinely engage with the questionnaire, often with more random response patterns as a result. Using a mixture IRT approach and a meta-analytic lens across 22 educational systems participating in TIMSS 2015, we investigated whether the prevalence of random responders on six scales regarding students’ engagement and attitudes toward mathematics and sciences was a function of grade, gender, socio-economic status, spoken language at home, or migration background. Among these common policy-relevant covariates in educational research, we found support for small group differences in prevalence of random responders (({text {OR}}ge 1.22)) (average prevalence of 7%). In general, being a student in grade 8 (vs. grade 4), being male, reporting to have fewer books, or speaking a language different from the test language at home were all risk factors characterizing random responders. The expected generalization and implications of these findings are discussed based on the observed heterogeneity across educational systems and consistency across questionnaire scales.