How representative is the Swedish PISA sample? A comparison of PISA and register data

IF 2.8 3区教育学 Q1 EDUCATION & EDUCATIONAL RESEARCH Educational Assessment Evaluation and Accountability Pub Date : 2024-07-06 DOI:10.1007/s11092-024-09438-5

Linda Borger, Stefan Johansson, Rolf Strietholt

{"title":"How representative is the Swedish PISA sample? A comparison of PISA and register data","authors":"Linda Borger, Stefan Johansson, Rolf Strietholt","doi":"10.1007/s11092-024-09438-5","DOIUrl":null,"url":null,"abstract":"<p>PISA aims to serve as a “global yardstick” for educational success, as measured by student performance. For comparisons to be meaningful across countries or over time, PISA samples must be representative of the population of 15-year-old students in each country. Exclusions and non-response can undermine this representativeness and potentially bias estimates of student performance. Unfortunately, testing the representativeness of PISA samples is typically infeasible due to unknown population parameters. To address this issue, we integrate PISA 2018 data with comprehensive Swedish registry data, which includes all students in Sweden. Utilizing various achievement measures, we find that the Swedish PISA sample significantly overestimates the achievement levels in Sweden. The observed difference equates to standardized effect sizes ranging from <i>d</i> = .19 to .28, corresponding to approximately 25 points on the PISA scale or an additional year of schooling. The paper concludes with a plea for more rigorous quality standards and their strict enforcement.</p>","PeriodicalId":46725,"journal":{"name":"Educational Assessment Evaluation and Accountability","volume":"85 1","pages":""},"PeriodicalIF":2.8000,"publicationDate":"2024-07-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Educational Assessment Evaluation and Accountability","FirstCategoryId":"95","ListUrlMain":"https://doi.org/10.1007/s11092-024-09438-5","RegionNum":3,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"EDUCATION & EDUCATIONAL RESEARCH","Score":null,"Total":0}

引用次数: 0

Abstract

PISA aims to serve as a “global yardstick” for educational success, as measured by student performance. For comparisons to be meaningful across countries or over time, PISA samples must be representative of the population of 15-year-old students in each country. Exclusions and non-response can undermine this representativeness and potentially bias estimates of student performance. Unfortunately, testing the representativeness of PISA samples is typically infeasible due to unknown population parameters. To address this issue, we integrate PISA 2018 data with comprehensive Swedish registry data, which includes all students in Sweden. Utilizing various achievement measures, we find that the Swedish PISA sample significantly overestimates the achievement levels in Sweden. The observed difference equates to standardized effect sizes ranging from d = .19 to .28, corresponding to approximately 25 points on the PISA scale or an additional year of schooling. The paper concludes with a plea for more rigorous quality standards and their strict enforcement.

查看原文

微信好友朋友圈 QQ好友复制链接

本刊更多论文

瑞典国际学生评估项目（PISA）样本的代表性如何？PISA 与登记数据的比较

国际学生评估项目旨在作为衡量教育成功与否的 "全球尺度"，以学生的表现来衡量。为了使各国之间或不同时期的比较具有意义，PISA 的样本必须能够代表每个国家 15 岁的学生群体。排除和不响应会破坏这种代表性，并可能使对学生成绩的估计出现偏差。遗憾的是，由于人口参数未知，测试 PISA 样本的代表性通常是不可行的。为了解决这个问题，我们将 PISA 2018 数据与瑞典的综合登记数据相结合，其中包括瑞典的所有学生。利用各种成绩衡量标准，我们发现瑞典 PISA 样本大大高估了瑞典的成绩水平。观察到的差异相当于 d = 0.19 至 0.28 的标准化效应大小，相当于 PISA 量表上的约 25 分或额外一年的学校教育。本文最后呼吁制定更严格的质量标准并严格执行。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文去求助

来源期刊

Educational Assessment Evaluation and Accountability EDUCATION & EDUCATIONAL RESEARCH-

CiteScore

9.40

自引率

2.60%

发文量

期刊介绍： The main objective of this international journal is to advance knowledge and dissemination of research on and about assessment, evaluation and accountability of all kinds and on various levels as well as in all fields of education. The journal provides readers with an understanding of the rich contextual nature of evaluation, assessment and accountability in education. The journal is theory-oriented and methodology-based and seeks to connect research, policy making and practice. The journal publishes outstanding empirical works, peer-reviewed by eminent scholars around the world.Aims and Scope in more detail: The main objective of this international journal is to advance knowledge and dissemination of research on and about evaluation, assessment and accountability: - of all kinds (e.g. person, programme, organisation), - on various levels (state, regional, local), - in all fields of education (primary, secondary, higher education/tertiary, as well as non-school sector) and across all different life phases (e.g. adult education/andragogy/Human Resource Management/professional development).The journal provides readers with an understanding of the rich contextual nature of evaluation, assessment and accountability in education. The journal is theory-oriented and methodology-based and seeks to connect research, policy making and practice. Therefore, the journal explores and discusses: - theories of evaluation, assessment and accountability, - function, role, aims and purpose of evaluation, assessment and accountability, - impact of evaluation, assessment and accountability, - methodology, design and methods of evaluation, assessment and accountability, - principles, standards and quality of evaluation, assessment and accountability, - issues of planning, coordinating, conducting, reporting of evaluation, assessment and accountability.The journal also covers the quality of different instruments or procedures or approaches which are used for evaluation, assessment and accountability.The journal only includes research findings from evaluation, assessment and accountability, if the design or approach of it is meta-reflected in the article.The journal publishes outstanding empirical works, peer-reviewed by eminent scholars around the world.