A critical evaluation of the validity of socioeconomic measures used in PISA

Abstract

Purpose

Given the scientific and practical difficulties inherent in measuring and comparing socioeconomic deprivation (SED), and the further complexity added in cross national measurements, the main aim of this paper was to check the validity of SED measures used in PISA 2018 dataset. The SED measure used in PISA 2018 was the PISA index of economic, social and cultural status abbreviated as the ESCS index. This assessment was important as PISA analysis is based on variables derived from this instrument and the ESCS index and these reports influence and reflect international and comparative education policies and practice.

Design/methodology/approach

This study critically evaluates the socioeconomic status measures in the PISA 2018 dataset, focusing on their convergent validity and cross-national comparability. Using responses from over 600,000 students in 73 countries, it examines the validity of SES indicators and their comparability across countries. The study employs principal component analysis to construct local SES measures and compares them with the existing Economic, Social, and Cultural Status (ESCS) index. It explores the relationship between these SES measures and academic achievement in reading, science, and mathematics, aiming to understand their predictive validity in diverse educational settings. Statistical analyses were conducted using the IEA’s IDB Analyser and SPSS, ensuring robustness and generalisability across the diverse participant countries.

Findings

Our research findings challenge the assumed superiority of local measures over broader constructs like the Economic, Social, and Cultural Status (ESCS). It suggests that standardised measures like ESCS may provide more reliable predictions of academic achievement across various educational contexts, underscoring the complex relationship between SES measures and academic performance.

Originality/value

Our novel analysis shows that local and cross-national SED measures are poorly correlated. Our findings raise questions about the measures' validity while acknowledging the methodological challenges. We provide empirical evidence to support ongoing debates on the topic.