Machine learning in public health informatics: Evidence that complex sampling structures may not be needed for prediction models with imbalanced outcomes

Abstract

Purpose

The objective of this study is to investigate the predictive ability of machine learning models for imbalanced outcomes from national survey data without the use of sampling weights.

Methods

We evaluated the predictive performance of machine learning models on imbalanced outcomes from the US National Health and Nutrition Examination Survey (USNHANES) without using sampling weights. Four machine learning models (support vector machine, random forest, least absolute shrinkage and selection operator regression, and deep neural network) were compared with a logistic model that incorporates the survey's complex sampling design. Three resampling methods (oversampling, undersampling, and combined) were used to address class imbalance during the model training process.

Results

For all models, the balanced accuracy was similar (ranging from 0.72 to 0.76) and the specificity was smaller than sensitivity except for random forest. The support vector machine and neural networks performed best with sensitivity (ranging from 0.79 to 0.83), while the random forest had the largest specificity (ranging from 0.86 to 0.96), with one exception. PR-AUC scores and Brier scores were low ranging from 0.2529 to 0.3313 (lower scores worse) and 0.1005–0.3245 (lower scores better), respectively

Conclusions

The machine learning models had overall similar predictive capacity to the recommended methods which integrate the complex sampling design for the prediction of osteoarthritis occurrence with USNHANES.