Assessing the representativeness of large medical data using population stability index.

Abstract

Background: Understanding sample representativeness is key to interpreting findings from epidemiological research and applying these findings to broader populations. Though techniques for assessing sample representativeness are available, they rely on access to raw data detailing the population of interest which are often not readily available and may not be suitable for comparing large datasets. In reality, population-based data are often only available in an aggregated format. In this study, we aimed to examine the capability of population stability index (PSI), a popular metric to assess data drift for artificial intelligence studies, in detecting sample differences using population-based data.

Method: We obtained United States cancer statistics from the National Cancer Institute's Surveillance, Epidemiology, and End Results (SEER) database. We queried the SEER 17-registry research database to obtain cancer count data by age, sex, and cancer site groups from the rate sessions of the SEER*State incidence database for 2000 and 2015 - 2020. We then calculated PSI scores to estimate yearly data distribution shift from 2015 to 2020 for each variable. We compared the PSI results to the Chi-Square and Cramér's V tests for the same comparisons.

Results: Scores for PSI comparing age, sex, and cancer site distribution between years ranged widely from 2.96 to less than 0.01. In line with our expectations, we found moderate to substantial differences in cancer population characteristics between 2000 and all other included years using PSI. Despite small effect sizes (Cramér's V 0.01 - 0.09), Chi-Square tests were significant for most comparisons, indicating likely type-I error caused by our large sample.

Conclusions: Population stability index can be used to examine sample differences in healthcare studies where only binned data are available or where large datasets may reduce the reliability of other metrics. Inclusion of PSI in epidemiological research will give greater confidence that results are representative of the general population.