Large-Scale Assessments in Education最新文献

To assess the integration of information and communication technologies (ICTs) in teaching and learning, a new construct for ICT competence was developed by the Programme for International Student Assessment (PISA) in 2022. The framework for ICT assessment involves four major dimensions; access to ICT, use of ICT, students’ ICT competencies, and subject-related ICT use. In this paper, we analysed this ICT assessment in terms of validity construction of test scores with data from PISA 2018 and investigated ICT competence in relation to the variables of mathematics, science, and reading literacy. The analyses were based on public data from PISA 2018 for Turkey (N = 6890). After removal of univariate and multivariate outliers from the Turkish sample, factor analysis, correlation analysis, and structural equation modelling were performed on 5438 data items. In the study, we confirmed the four-dimensional structure of ICT competence for the Turkish sample. In this study, the construct validity of the PISA 2018 application ICT test was verified using the Turkish sample. The ESCS variable was included as a control variable in the analyses due to its significant impact on the relevant variables. The analysis showed that the mathematical literacy scores of the PISA 2018 Turkish students were positively low affected by the availability of ICT, negatively low affected by the entertainment of ICT, and positively low affected by the interest and perceived competence in ICT, use of ICT for educational purposes at school and outside school, and the use of ICT related to the subject. Considering the ICT variables that affected the PISA 2018 students’ science literacy scores; the entertainment of ICT and the interest and perceived competence in ICT significantly low affected the students’ use of ICT on the subject. On the other hand, the availability of ICT negative significantly low affect their science literacy scores. While the entertainment of ICT negatively low affected the science literacy scores, interest and perceived competence in ICT and use of ICT for educational purposes at school and outside school positively low affected the students’ use of ICT on the subject. Finally, when we look at the ICT variables that low affected the students’ reading literacy scores; and use of ICT for educational purposes at school and outside school and the interest and perceived competence in ICT significantly positive low affected the use of ICT on the subject. On the other hand, the availability of ICT negative significantly low affect reading literacy scores. While the entertainment of ICT negatively low affected reading literacy scores; interest and perceived competence in ICT positively low affected the use of ICT on the subject.

This paper extends existing work on teacher weighting in student-centered surveys by looking into aspects of practical implementation of deriving and using weights for teacher-centered analysis in the Trends in International Mathematics and Science Study (TIMSS) and the Progress in International Reading Literacy Study (PIRLS). The formal conditions to compute teacher-centered weights are detailed, including mathematical equations. We provide a proposal on how to define the targeted populations as well as how to collect data that is needed to derive teacher-centered weights, yet currently unavailable. We also tackle the issue of teacher nonresponse by proposing a respective adjustment factor, as well as mentioning the challenge of multiple selection probabilities when teachers teach in multiple schools. The core part of the paper focuses on studying the level of accuracy that can be expected when estimating teacher population characteristics. We use TIMSS 2019 data and simulate likely scenarios regarding the variance in weights. The results show that (i) the different weighting scenarios lead to relatively similar estimates; however, the differences between the scenarios are sufficient to justify the recommendation to use correctly derived teacher weights; (ii) differences between estimated standard errors based on complex sampling and corresponding estimates based on simple random sampling are sufficiently consistent to support use of a procedure to estimate standard errors that accounts for both sample weights and the complex sampling design; (iii) sample sizes and variance in weights significantly limit estimate precision, so that total population estimates with sufficient precision are available in the majority of countries but subpopulation features are generally not sufficiently precise. To provide a critical evaluation of our results, we recommend implementation of the proposed method in one or more countries. This recommended study will permit examination of logistical considerations in implementation of required changes in data acquisition and will provide data to replicate the analysis with teacher-centered weights.

Continued migration flows are inevitable, and research into favorable conditions for educating students in pluralistic societies is timely. This study attempts to address the operationalization of favorable conditions when educating children in diverse societies. Previous research into the contact hypothesis and acculturation theory has suggested that moderating factors are necessary for contact between groups of “others” to effect positive change in attitudes of tolerance and reduce prejudice. To represent groups of “others”, this study—which analyzed ICCS data from both the 2009 and 2016 cycles—considers lower-secondary students with an immigration background, who speak a different language than the administered assessment in the home, and diversity in terms of socio-economic background. To represent a desirable or “successful” classroom context, the study focuses on attitudes of tolerance held by lower-secondary students towards minorities, investigating whether participation in civic activities in the school or community, or the act of learning in more diverse classrooms, was associated with increased rates of tolerance. Findings supported the stated hypotheses to some extent, as some associations between heterogenous classroom composition (in terms of the share of students from immigrant backgrounds and those who spoke different languages in the home) and increased attitudes of tolerance towards minorities were observed. Student participation in civic activities in school was significantly associated with higher attitudes of tolerance towards minorities in 16 out of 18 analyzed countries. A relation between civic participation in the community and tolerance towards minorities was not found. Overall, students’ rates of tolerance towards minorities increased significantly for the majority of analyzed countries between 2009 and 2016.

Because large-scale studies repeatedly indicated low reading literacy for many students, a need for interventions fostering reading literacy, such as extracurricular tutoring, has often been emphasized. Several reading promoting programs, suitable for extracurricular tutoring, were developed and shown to be effective in recent years. Moreover, these studies and analyses of extracurricular activities and tutoring yielded findings on learning-supporting characteristics and effects on students’ achievement of such extracurricular offers. Nevertheless, little is known to date about the implementation of extracurricular tutoring in practice in Germany, particularly about its participation rates, characteristics, and effects on students` reading literacy. Thus, the present study investigated participation rates of extracurricular tutoring in reading and in German among students in general and among various subgroups between Grades 5 and 7. Furthermore, the study examined general and subject-specific characteristics and effects of extracurricular tutoring on reading literacy. The analyses used a sample of students (N = 5113) of the National Educational Panel Study in Germany. In addition to descriptive analyses of characteristics and participation rates of extracurricular tutoring, effect sizes were calculated for comparing participation rates of subgroups of students. Furthermore, multi-group structural equation models were implemented to investigate average and differential effects of extracurricular tutoring, while controlling for initial reading literacy and other covariates. The results indicate that mainly students with low reading literacy and therefore a strong need for support participated in extracurricular tutoring, but the general participation rate was low. The descriptive outcomes showed a heterogeneous picture of tutoring offers and rare implementations of reading-promoting methods. Moreover, the results revealed no significant effects of extracurricular tutoring on students’ reading literacy overall but suggested marginal differences for students with a low initial reading literacy. For future studies, more detailed assessments of characteristics and methods of extracurricular tutoring are recommended, particularly, in large-scale investigations on specific tutoring methods for students with support needs which may lead to improved guidelines regarding promising implementations of evidence-based reading promoting programs.

The study of variables related to the use of ICT in the classroom is a topic of interest that has been frequently researched. In this paper, after examining the importance of teacher training in explaining the use of ICT in the classroom, we focused on analysing the weight of variables related to teaching practices and the organisational context of schools, which are variables that are less frequently addressed in the literature. To do so, a secondary analysis was carried out using data provided by the Teaching and Learning International Study (TALIS 2018). Specifically, we worked with a sample of 3,918 principals and 64,899 teachers from a total of 3921 schools in 21 countries. A multilevel binary regression model with random intercept, fixed coefficients and a two-level structure with teachers at level 1 and schools at level 2 was used. The results indicate that the presence of ICT in the classroom is associated with self-efficacy in teaching and the cognitive activation of students and with the organisational aspects of the school, which are scarcely addressed by the existing literature on this topic of interest, such as school climate, educational innovation and cooperation among teachers. Based on these results, we reflect on possible ways to promote the use of ICT in the classroom.

Recent years have witnessed signs of increasing political instability in democratic countries as well as growing alienation from civic institutions and processes among citizens, especially among young people. Within the context of civic and citizenship education, it is important to review such phenomena and study their extent among young people as well as the factors that have the potential of promoting different forms of citizenship engagement. Using data from the International Civic and Citizenship Education Study (ICCS) 2016 and 2009, this article provides insights into the expectations of young people to actively engage as citizens in the future and what influences these expectations, with a primary focus on the role of civic knowledge and trust in civic institutions.

Results from ICCS 2009 and 2016 show that while large majorities among young people expected to vote in elections, only relatively few found it likely to be more actively involved in political action. Except for engagement in illegal protest, young people’s expected participation in general appeared to be positively related to trust. However, associations with civic knowledge were more differentiated. Trust and civic knowledge tended to have negative correlations in countries with higher levels of perceived corruption, while a different association became apparent in democracies with more transparent institutions. Civic knowledge was consistently positively related to anticipated voting while it was negatively related to expected illegal protest. More knowledgeable students were also less inclined to consider active (conventional) forms of political participation in the future.

Background

Previous research based on large-scale studies consistently suggests that on average, male students tend to have lower literacy compared to their female students during secondary schooling. However, this gender gap in literacy seems to “disappear” during adulthood. Up until today, only a few studies investigated the role of assessment features in gender differences in literacy performance in adulthood. This study aims to understand the relationship between assessment features and gender differences in literacy skills.

Methods

Using the German 2012 PIAAC data (N = 4,512), we applied item-level analyses using linear probability models to examine gender differences in the probability of solving a literacy item correctly with six assessment features including (1) text format, (2) text topics, (3) text length, (4) cognitive strategies, (5) numerical content of the text/questions, and (6) gender typicality of content.

Results

We found that men had a 13.4% higher probability of solving items with a noncontinuous text format correctly than women. Men also had a 9.4% higher probability of solving short text items correctly and a 4.6% higher probability of solving items with a medium/high numerical content in the question correctly than women. There were small to negligible gender differences in literacy performance in terms of text topics, cognitive strategies, and gender typicality of content.

Conclusions

Our findings highlight the role of text format, text length, and numerical content in gender differences in literacy skills, suggesting that further refining these practices can enhance the fairness and accuracy of literacy assessments. Specifically, we advocate for ongoing research aimed at understanding and minimizing the potential bias introduced by these assessment features. Such efforts are not only crucial for developing instruments that accurately measure literacy skills, but they also yield insights that hold significant implications for educational researchers and practitioners dedicated to creating more equitable assessment environments.

Background

This empirical study aims to investigate the association between mathematics self-efficacy and mathematics achievement gaps among students in Grades 4, 8, and 12, utilizing data from the 2019 National Assessment of Educational Progress (NAEP). The study also considers student-level (e.g., mathematics self-efficacy, gender, race/ethnicity) and school-level (e.g., school location, proportion of underrepresented students) demographics to provide a comprehensive analysis of the factors contributing to mathematics achievement gaps.

Methods

A two-level cross-sectional multilevel modeling approach was employed to analyze the variance in mathematics achievement, partitioning it into within- and between-school components. This approach allowed for an examination of association between mathematics self-efficacy and achievement gaps while considering various student and school characteristics. The rationale behind this methodology lies in its ability to capture the hierarchical nature of educational data and provide a nuanced understanding of the factors associated with mathematics achievement.

Results

The analysis of the NAEP data revealed substantial variability in mathematics achievement across schools in the United States at all grade levels. Furthermore, mathematics self-efficacy emerged as a robust predictor of students' mathematics achievement, exhibiting significant effect sizes for Grades 4, 8, and 12. Remarkably, when students' mathematics self-efficacy was held constant, the mathematics achievement gaps among different student subgroups by gender, race/ethnicity, ELL, IEP, NSLP status narrowed, highlighting the importance of self-efficacy in addressing these disparities. The study also identified the presence of significant school contextual effects, further emphasizing the role of the educational environment in shaping mathematics achievement.

Conclusions

This study underscores the critical role of mathematics self-efficacy in influencing mathematics achievement gaps among students. By acknowledging the association between self-efficacy beliefs and mathematics achievement, policymakers and educators can develop targeted interventions to enhance students' confidence and motivation in mathematics, ultimately promoting equitable educational outcomes. The findings also emphasize the significance of school-level factors, calling for comprehensive approaches that consider both individual and contextual factors in narrowing achievement gaps. The implications of adopting a self-efficacy perspective to address mathematics achievement gaps extend to educational policy, curriculum development, and instructional practices, ultimately fostering more inclusive and effective mathematics education for all students.

Stratification is an important design feature of many studies using complex sampling designs and it is often used in large-scale assessment (LSA) studies, such as the Programme for International Student Assessment (PISA), for two main reasons. First, stratification variables that achieve a high between and low within strata variance can improve the efficiency of a survey design. Second, stratification allows one to, explicitly or implicitly, control for sample sizes across subpopulations. It ensures that some parts of a population are in the sample in predetermined proportions. In this study, we determine through simulation which stratification scheme is best for PISA in Germany. For this, we consider the constraints imposed by the international sampling design, the available information about schools, and specific national characteristics of the German educational system. We examine seven different stratification designs selected based on scenarios used in past LSAs in Germany and theoretical considerations for future implementations. The chosen scenarios were compared with two reference scenarios: (1) an unstratified design and (2) a synthetic optimal stratification design. The simulation study reveals that the stratification design currently applied in PISA produces satisfactory results regarding sampling precision. The present stratification design is based on Germany's federal states and school types. However, this approach leads to small strata, which has been problematic for estimating sampling variance in previous cycles. Therefore, alternative stratification scenarios were considered and, in addition to overcoming the small-strata problem, also led to smaller standard errors for estimates of student mean performance in mathematics, science, and reading. As a result of this study, we recommend considering three different stratification designs for Germany in future cycles of PISA. These recommendations aim to: (1) improve the sampling efficiency while keeping the sample size constant, (2) follow a sound methodological approach, and (3) make conservative and cautious changes while maintaining a reflection of the structure of the German federal school system with different school types. These suggestions include a reinvented stratification of grouped German federal states and designs with school types as explicit stratifiers and federal states as implicit stratifiers.

Self-efficacy or perceived ability refers to the confidence an individual has in their ability to successfully perform a specific task. Previous researchers have demonstrated the importance of self-efficacy beliefs and its impact on predicting students’ mathematics performances. Specifically, gender has been regarded as an important moderator variable in the prediction of achievement from measures of affect. To this end, using data from High School Longitudinal Study of 2009, this study utilized the multi-group analysis under the framework of structural equation modeling to examine gender differences in the reciprocal relationship between mathematics self-efficacy and mathematics achievement. The results showed that there were statistically significant differences between male students and female students in the reciprocal relationship between mathematics self-efficacy and mathematics achievement; the impact of prior mathematics achievement on later mathematics achievement was stronger than the impact of prior mathematics self-efficacy on later mathematics self-efficacy. Significance of these findings were discussed in this paper.