Pub Date : 2023-12-06DOI: 10.1007/s11092-023-09422-5
Sabine Meinck, Jörg-Henrik Heine, Julia Mang, Gabriel Nagy
This study uses evidence from a longitudinal survey (PISA Plus, Germany) to examine the potential of bias in international large-scale assessments (ILSAs). In PISA Plus, participation was mandatory at the first measurement point, but voluntary at the second measurement point. The study provides evidence for relevant selection bias regarding student competencies and background variables when participation is voluntary. Sample dropout at the second measurement point was related to characteristics such as family background, achievement in mathematics, reading and science, and other student and school demographic variables at both the student and school levels, with lower performing students and those with less favorable background characteristics having higher dropout frequencies, from which higher dropout probabilities of such students can be inferred. We further contrast the possibilities for addressing non-response through weight adjustments in longitudinal surveys with those in cross-sectional surveys. Considering our results, we evaluate and confirm the validity and appropriateness of strict participation rate requirements in ILSAs. Likely magnitudes of bias in cross-sectional studies in varying scenarios are illustrated. Accordingly, if combined participation rates drop below 70%, a difference of at least one-fifth of a standard deviation in an achievement score between non-respondents and participants leads to relevant bias. When participation drops below 50%, even a very small difference (one-tenth of a standard deviation) will cause non-negligible bias. Finally, we conclude that the stringent participation rate requirements established in most ILSAs are fully valid, reasonable, and important since they ensure a relatively low risk of biased results.
本研究利用一项纵向调查（PISA Plus，德国）的证据来研究国际大规模评估（ILSA）中可能存在的偏差。在 PISA Plus 中，第一个测量点的参与是强制性的，而第二个测量点的参与则是自愿的。研究证明，在自愿参与的情况下，学生的能力和背景变量会出现相关的选择偏差。在学生和学校层面，第二个测量点的样本辍学与家庭背景、数学、阅读和科学成绩以及其他学生和学校人口统计变量等特征有关，成绩较差的学生和背景特征较差的学生辍学频率较高，由此可以推断这些学生的辍学概率较高。我们进一步对比了纵向调查和横截面调查中通过权重调整解决非响应问题的可能性。考虑到我们的结果，我们评估并确认了在 ILSA 中严格要求参与率的有效性和适当性。我们还说明了不同情况下横断面研究中可能出现的偏差幅度。因此，如果综合参与率低于 70%，则未参与调查者和参与调查者的成绩得分至少相差五分之一个标准差，就会导致相关偏差。当参与率降至 50%以下时，即使是很小的差异（十分之一的标准差）也会造成不可忽视的偏差。最后，我们得出结论，大多数国际语言能力评估中规定的严格的参与率要求是完全有效、合理和重要的，因为它们确保了结果出现偏差的风险相对较低。
{"title":"Bias risks in ILSA related to non-participation: evidence from a longitudinal large-scale survey in Germany (PISA Plus)","authors":"Sabine Meinck, Jörg-Henrik Heine, Julia Mang, Gabriel Nagy","doi":"10.1007/s11092-023-09422-5","DOIUrl":"https://doi.org/10.1007/s11092-023-09422-5","url":null,"abstract":"<p>This study uses evidence from a longitudinal survey (PISA Plus, Germany) to examine the potential of bias in international large-scale assessments (ILSAs). In PISA Plus, participation was mandatory at the first measurement point, but voluntary at the second measurement point. The study provides evidence for relevant selection bias regarding student competencies and background variables when participation is voluntary. Sample dropout at the second measurement point was related to characteristics such as family background, achievement in mathematics, reading and science, and other student and school demographic variables at both the student and school levels, with lower performing students and those with less favorable background characteristics having higher dropout frequencies, from which higher dropout probabilities of such students can be inferred. We further contrast the possibilities for addressing non-response through weight adjustments in longitudinal surveys with those in cross-sectional surveys. Considering our results, we evaluate and confirm the validity and appropriateness of strict participation rate requirements in ILSAs. Likely magnitudes of bias in cross-sectional studies in varying scenarios are illustrated. Accordingly, if combined participation rates drop below 70%, a difference of at least one-fifth of a standard deviation in an achievement score between non-respondents and participants leads to relevant bias. When participation drops below 50%, even a very small difference (one-tenth of a standard deviation) will cause non-negligible bias. Finally, we conclude that the stringent participation rate requirements established in most ILSAs are fully valid, reasonable, and important since they ensure a relatively low risk of biased results.</p>","PeriodicalId":46725,"journal":{"name":"Educational Assessment Evaluation and Accountability","volume":null,"pages":null},"PeriodicalIF":3.9,"publicationDate":"2023-12-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"138545528","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-11-22DOI: 10.1007/s11092-023-09418-1
Pablo Bezem
Test-based accountability offers schools limited guidance on informing sustained reforms, whereas on-site school evaluations provide broader and more nuanced evidence. In order to transform inspection feedback into long-term reforms, school leaders’ attitudes towards these evaluations are critical. Relying on organizational change theory, this study examines principals’ beliefs regarding school inspection feedback effectiveness and appropriateness as well as their readiness to implement changes. A large US district that has experimented with such school inspection serves as the case study. The results demonstrate the principals’ strong positive attitudes towards inspection, and these attitudes are associated with sustained change. But, because the accountability framework did not center on inspection, principals were ambivalent about implementing specific changes. This study highlights the tensions involved in incorporating on-site, expert evaluations in test-based accountability systems.
{"title":"Principals’ attitudes towards on-site school evaluations: contribution to sustained school reform","authors":"Pablo Bezem","doi":"10.1007/s11092-023-09418-1","DOIUrl":"https://doi.org/10.1007/s11092-023-09418-1","url":null,"abstract":"<p>Test-based accountability offers schools limited guidance on informing sustained reforms, whereas on-site school evaluations provide broader and more nuanced evidence. In order to transform inspection feedback into long-term reforms, school leaders’ attitudes towards these evaluations are critical. Relying on organizational change theory, this study examines principals’ beliefs regarding school inspection feedback effectiveness and appropriateness as well as their readiness to implement changes. A large US district that has experimented with such school inspection serves as the case study. The results demonstrate the principals’ strong positive attitudes towards inspection, and these attitudes are associated with sustained change. But, because the accountability framework did not center on inspection, principals were ambivalent about implementing specific changes. This study highlights the tensions involved in incorporating on-site, expert evaluations in test-based accountability systems.</p>","PeriodicalId":46725,"journal":{"name":"Educational Assessment Evaluation and Accountability","volume":null,"pages":null},"PeriodicalIF":3.9,"publicationDate":"2023-11-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"138516348","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-10-23DOI: 10.1007/s11092-023-09421-6
Luis Alejandro Lopez-Agudo, Claudia Prieto Latorre, Oscar David Marcenaro-Gutierrez
{"title":"Grade retention in Spain: the right way?","authors":"Luis Alejandro Lopez-Agudo, Claudia Prieto Latorre, Oscar David Marcenaro-Gutierrez","doi":"10.1007/s11092-023-09421-6","DOIUrl":"https://doi.org/10.1007/s11092-023-09421-6","url":null,"abstract":"","PeriodicalId":46725,"journal":{"name":"Educational Assessment Evaluation and Accountability","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2023-10-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"135365518","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-10-18DOI: 10.1007/s11092-023-09420-7
Cherry Zin Oo, Dennis Alonzo, Ria Asih, Giovanni Pelobillo, Rex Lim, Nang Mo Hline San, Sue O’Neill
{"title":"Implementing school-based assessment reforms to enhance student learning: a systematic review","authors":"Cherry Zin Oo, Dennis Alonzo, Ria Asih, Giovanni Pelobillo, Rex Lim, Nang Mo Hline San, Sue O’Neill","doi":"10.1007/s11092-023-09420-7","DOIUrl":"https://doi.org/10.1007/s11092-023-09420-7","url":null,"abstract":"","PeriodicalId":46725,"journal":{"name":"Educational Assessment Evaluation and Accountability","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2023-10-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"135888592","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-10-12DOI: 10.1007/s11092-023-09419-0
Xumei Fan
{"title":"Accountability in the evaluation of teacher effectiveness: views of teachers and administrators","authors":"Xumei Fan","doi":"10.1007/s11092-023-09419-0","DOIUrl":"https://doi.org/10.1007/s11092-023-09419-0","url":null,"abstract":"","PeriodicalId":46725,"journal":{"name":"Educational Assessment Evaluation and Accountability","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2023-10-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"136014381","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-09-30DOI: 10.1007/s11092-023-09415-4
Alec I. Kennedy, Rolf Strietholt
Abstract The COVID-19 pandemic disrupted education worldwide as educational systems made the decision to close schools to contain the spread of the virus. The duration of school closures varied greatly internationally. In this study, we use international variation in school closure policies to examine the effects of school closures on student achievement. Specifically, we use representative trend data from more than 300,000 students in 29 countries to examine whether the length of school closures is related to changes in student achievement before and after the outbreak of COVID-19. We observe a significant and substantial negative effect of school closures on student reading achievement. This school closure effect remains even after controlling for measures of pandemic severity such as infection rates, vaccination policies, and a measure of lockdown stringency. The estimated effect implies that a year of school closures corresponds roughly to the loss of a little more than half a school year of learning. This effect is even more pronounced for socioeconomically disadvantaged students and those without home computer access.
{"title":"School closure policies and student reading achievement: evidence across countries","authors":"Alec I. Kennedy, Rolf Strietholt","doi":"10.1007/s11092-023-09415-4","DOIUrl":"https://doi.org/10.1007/s11092-023-09415-4","url":null,"abstract":"Abstract The COVID-19 pandemic disrupted education worldwide as educational systems made the decision to close schools to contain the spread of the virus. The duration of school closures varied greatly internationally. In this study, we use international variation in school closure policies to examine the effects of school closures on student achievement. Specifically, we use representative trend data from more than 300,000 students in 29 countries to examine whether the length of school closures is related to changes in student achievement before and after the outbreak of COVID-19. We observe a significant and substantial negative effect of school closures on student reading achievement. This school closure effect remains even after controlling for measures of pandemic severity such as infection rates, vaccination policies, and a measure of lockdown stringency. The estimated effect implies that a year of school closures corresponds roughly to the loss of a little more than half a school year of learning. This effect is even more pronounced for socioeconomically disadvantaged students and those without home computer access.","PeriodicalId":46725,"journal":{"name":"Educational Assessment Evaluation and Accountability","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2023-09-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"136341938","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-09-27DOI: 10.1007/s11092-023-09417-2
Eli Jones, Christi Bergin, Bridget Murphy
{"title":"Correction to: Principals may inflate teacher evaluation scores to achieve important goals","authors":"Eli Jones, Christi Bergin, Bridget Murphy","doi":"10.1007/s11092-023-09417-2","DOIUrl":"https://doi.org/10.1007/s11092-023-09417-2","url":null,"abstract":"","PeriodicalId":46725,"journal":{"name":"Educational Assessment Evaluation and Accountability","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2023-09-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"135537280","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-09-21DOI: 10.1007/s11092-023-09416-3
Umut Atasever, John Jerrim, Sabine Tieck
Abstract Cross-national comparisons of educational achievement rely upon each participating country collecting nationally representative data. While obtaining high response rates is a key part of reaching this goal, other potentially important factors may also be at play. This paper focuses on one such issue—exclusion rates—which has received relatively little attention in the academic literature. Using data from 20 years of international large-scale assessment data, we find there to be modest variation in exclusion rates across countries and that there has been a relatively small increase in exclusion rates in some over time. We also demonstrate how exclusion rates tend to be higher in studies of primary students than in studies of secondary students. Finally, while there seems to be little relationship between exclusion rates and response rates, there is a weak negative association between the level of exclusions and test performance. We conclude by discussing how information about exclusions—and other similar issues—might be more clearly communicated to non-specialist audiences.
{"title":"Exclusion rates from international large-scale assessments: an analysis of 20 years of IEA data","authors":"Umut Atasever, John Jerrim, Sabine Tieck","doi":"10.1007/s11092-023-09416-3","DOIUrl":"https://doi.org/10.1007/s11092-023-09416-3","url":null,"abstract":"Abstract Cross-national comparisons of educational achievement rely upon each participating country collecting nationally representative data. While obtaining high response rates is a key part of reaching this goal, other potentially important factors may also be at play. This paper focuses on one such issue—exclusion rates—which has received relatively little attention in the academic literature. Using data from 20 years of international large-scale assessment data, we find there to be modest variation in exclusion rates across countries and that there has been a relatively small increase in exclusion rates in some over time. We also demonstrate how exclusion rates tend to be higher in studies of primary students than in studies of secondary students. Finally, while there seems to be little relationship between exclusion rates and response rates, there is a weak negative association between the level of exclusions and test performance. We conclude by discussing how information about exclusions—and other similar issues—might be more clearly communicated to non-specialist audiences.","PeriodicalId":46725,"journal":{"name":"Educational Assessment Evaluation and Accountability","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2023-09-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"136235650","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-08-24DOI: 10.1007/s11092-023-09412-7
Kseniia Marcq, J. Braeken
{"title":"Gender differences in item nonresponse in the PISA 2018 student questionnaire","authors":"Kseniia Marcq, J. Braeken","doi":"10.1007/s11092-023-09412-7","DOIUrl":"https://doi.org/10.1007/s11092-023-09412-7","url":null,"abstract":"","PeriodicalId":46725,"journal":{"name":"Educational Assessment Evaluation and Accountability","volume":null,"pages":null},"PeriodicalIF":3.9,"publicationDate":"2023-08-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"42982171","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-08-01DOI: 10.1007/s11092-023-09414-5
G. Skedsmo, S. Huber
{"title":"Analyses of quality assurance and development in education","authors":"G. Skedsmo, S. Huber","doi":"10.1007/s11092-023-09414-5","DOIUrl":"https://doi.org/10.1007/s11092-023-09414-5","url":null,"abstract":"","PeriodicalId":46725,"journal":{"name":"Educational Assessment Evaluation and Accountability","volume":null,"pages":null},"PeriodicalIF":3.9,"publicationDate":"2023-08-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"42459824","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
扫码关注我们
求助内容：
应助结果提醒方式：
京公网安备 11010802042870号