A Multimetric Evaluation of Stratified Random Sampling for Classification: A Case Study

Accurate classification of biological phenotypes is an essential task for medical decision making. The selection of subjects for classifier training and validation sets is a crucial step within this task. To evaluate the impact of two approaches for subject selection—randomization and clinical balancing, we applied six classification algorithms to a highly replicated publicly available breast cancer data set. Using six performance metrics, we demonstrate that clinical balancing improves both training and validation performance for all methods on average. We also observed a smaller discrepancy between training and validation performance. Furthermore, a simple analytical argument is presented which suggests that we need only two metrics from the class of metrics based on the entries of the confusion matrix. In light of our results, we recommend: 1) clinical balancing of training and validation data to improve signal-to-noise ratio and 2) the use of multiple classification algorithms and evaluation metrics for a comprehensive evaluation of the decision making process.