Large-Scale Assessments in Education最新文献

Employing nested sequences of models is a common practice when exploring the extent to which one set of variables mediates the impact of another set. Such an analysis in the context of logistic regression models confronts two challenges: (i) direct comparisons of coefficients across models are generally biased due to the changes in scale that accompany the changes in the set of explanatory variables, (ii) conducting a large number of tests induces a problem of multiplicity that can lead to spurious findings of significance if not heeded. This article aims to illustrate a practical strategy for conducting analyses in the face of these challenges. The challenges—and how to address them—are illustrated using a subset of the findings reported by Braun (Large-scale Assess Educ 6(4):1–52, 2018. 10.1186/s40536-018-0058-x), drawn from the Programme for the International Assessment of Adult Competencies (PIAAC), an international, large-scale assessment of adults. For each country in the dataset, a nested pair of logistic regression models was fit in order to investigate the role of Educational Attainment and Cognitive Skills in mediating the impact of family background and demographic characteristics on the location of an individual’s annual income in the national income distribution. A modified version of the Karlson–Holm–Breen (KHB) method was employed to obtain an unbiased estimate of the true differences in the coefficients between nested logistic models. In order to address the issue of multiplicity, a recent generalization of the Benjamini–Hochberg (BH) False Discovery Rate (FDR)-controlling procedure to hierarchically structured hypotheses was employed and compared to two conventional methods. The differences between the changes in coefficients calculated conventionally and with the KHB adjustment varied from negligible to very substantial. When combined with the actual magnitudes of the coefficients, we concluded that the more proximal factors indeed act as strong mediators for the background factors, but less so for Age, and hardly at all for Gender. With respect to multiplicity, applying the FDR-controlling procedure yielded results very similar to those obtained by applying a standard per-comparison procedure, but quite a few more discoveries in comparison to the Bonferroni procedure. The KHB methodology illustrated here can be applied wherever there is interest in comparing nested logistic regressions. Modifications to account for probability sampling are practicable. The categorization of variables and the order of entry should be determined by substantive considerations. On the other hand, the BH procedure is perfectly general and can be implemented to address multiplicity issues in a broad range of settings.

The present study aimed to examine the relations of early numeracy activities and skills to mathematics dispositions, engagement, and achievement among 26,859 fourth graders in the United Arab Emirates who took part in the sixth cycle of the Trends in International Mathematics and Science Study (TIMSS) in 2015. The study also explored the mediating effects of mathematics dispositions and engagement on the relations between early numeracy activities and skills and mathematics achievement among these fourth graders. Results of path analyses, after controlling for participants’ demographic and socioeconomic characteristics, indicated that early numeracy activities and skills were significantly and positively related to mathematics dispositions, engagement, and achievement. Further, results of mediational analyses suggested that confidence in mathematics had a significant mediating effect on the relations between early numeracy activities and skills and mathematics achievement. The findings of the study highlight the crucial role that early numeracy activities and skills play in enhancing fourth graders’ mathematics dispositions, engagement, and achievement in the United Arab Emirates.

The study of the territorial difference in educational achievement is a widely debated topic, in particular in Italy for the presence of the well-known strong regional disparities. National and international large scale assessments confirmed that the main characteristic of the Italian school system is the geographical cleavage between North and South. Policymakers have pressing needs to find solutions to reduce geographical disparities. In this study, we investigate the spatial disparities of academic achievement from a new perspective, assuming that the relationship between academic achievement and predictors varies across Italy. Our aim is to examine the extent of the spatial disparities in the relationship between academic achievement and some school-level factors related to inequalities in educational outcomes, moving beyond the regional administrative confines, in order to identify new spatial patterns. We exploited the reading standardized tests administered by INVALSI in 2018–2019 focusing on the 8th-grade students. Crucial to our contribution is the use of the geographically weighted regression and the k-mean clustering, which allows studying the spatial variability of the impact of the school-level factors on academic achievement and to gather schools in new spatial clusters. The findings of this paper demonstrate the necessity to design a more specific education policy and support the identification of the main critical factors for different geographical areas.

Background

The idea of using low-stakes assessment results is often mentioned when designing educational system reforms. However, when tests have no consequences for the students, test takers may not make enough effort when completing the test, and their lack of engagement may negatively affect the validity of the conclusions of the studies that use such tests. This article presents analyses of student test-taking engagement in a low-stakes international large-scale assessment, the Programme for International Student Assessment (PISA), and (i) quantifies test-taking effort, (ii) predicts effort by means of expectancy-value proxies and (iii) investigates the relationship between effort and science proficiency.

Methods

Students’ response times on the science items of PISA 2015 were used to derive an index of test-taking effort. Data from six English-speaking countries that administered the computer-based version of the test, were selected. The response time for each item was modelled by means of a two-class finite mixture model, and students’ probabilities of being classified as effortful were combined to derive a global index of effort.

Results

Our findings showed that students’ effort decreased towards the end of the testing sessions. The variance of examinees’ test-taking effort was not substantially explained by the expectancy-value variables. Test-taking effort had a strong relationship with science achievement, with the correlation increasing to more than 0.5 towards the end of the test. Moreover, an important part of the relationship between test-taking effort and achievement is not related to other student characteristics, such as gender, socio-economic and cultural status, attitude towards school or attitude towards science.

Conclusions

This study shows that students put different amounts of effort into test taking, especially towards the end of the assessment, and suggests a possible underestimation of related student achievement that may affect the interpretation of test scores.

This paper introduces a new method to measure school-level inequality based on Item Response Theory (IRT) models. Categorical data collected by large-scale assessments poses diverse methodological challenges hinder measuring inequality due to data truncation and asymmetric intervals between categories. I use family possessions data from PISA 2015 to exemplify the process of computing the measurement and develop a set of country-level mixed-effects linear regression models comparing the predictive performance of the novel inequality measure with school-level Gini coefficients. I find school-level inequality is negatively associated with learning outcomes across many non-European countries.

Background

Studies demonstrate that occupational optimism can boost adolescents’ academic attainment and perseverance in education. To contribute to this literature, we consider two hypotheses. The first posits that bilingual immigrants are remarkably resourceful and determined. Thus, they are more occupationally ambitious than their peers. The second proposes that immigrant students engage in “strategic adaptation” by specializing in science, viewed as a level playing field.

Methods

To assess these hypotheses at two points of time, we analyze data from 19 societies that participated in the Programme for International Student Assessment (PISA) in 2006 and 2015. Our primary method is path analysis with balanced replicate weights (BRR) undertaken separately for each country’s data.

Results

We find that, in many countries, bilingual immigrants expect to enter higher status occupations than non-immigrants. However, immigrants who do not speak another language are also optimistic, so linguistic resources cannot explain occupational ambition. Furthermore, immigrants accord science more instrumental value and enjoy it more at school, which accounts, across societies, for up to 12% of the variation in vocational optimism indicated by the expected occupational status, and up to 41% in plans to pursue a career in science professions.

Conclusion

Our results align with the “strategic adaptation” argument that many young immigrants might seek to specialize in science as a pragmatic tactic to ensure high occupational attainment.

Background: Evidence so far shows that Waldorf students are characterized by average science achievement but at the same time high socioeconomic status and high science motivation. Moreover, Waldorf education is characterized by high emphasis on inquiry-based science education (IBSE). The present study investigates if the specific attitude-achievement constellation of Waldorf students in science may be explained by the high level of IBSE.

Methods: Propensity score matching was applied to control for Waldorf students' more advantageous social background using the Austrian PISA 2015 sample (N = 7007 15-year old students). After propensity score matching, 1107 matched controls were included alongside all 149 Waldorf students.

Results: The results show that Waldorf students report higher enjoyment in learning science and more interest in broad science topics than matched controls. On the other hand, they demonstrate lower science achievement. Mediation analyses show that, although Waldorf students' more positive attitudes towards science can be almost entirely attributed to their higher exposure to IBSE, their underperformance in science cannot.

Conclusions: These results indicate that attending a school type with a high level of IBSE (Waldorf) may have positive effects on attitudinal outcomes (enjoyment and interest in science) whereas it does not seem to have notable effects on science achievement. This indicates that IBSE could be applied in educational contexts aiming to increase students' scientific attitudes.