Fast Preliminary Evaluation of New Machine Learning Algorithms for Feasibility

Abstract

Traditionally, researchers compare the performance of new machine learning algorithms against those of locally executed simulations that serve as benchmarks. This process requires considerable time, computation resources, and expertise. In this paper, we present a method to quickly evaluate the performance feasibility of new algorithms – offering a preliminary study that either supports or opposes the need to conduct a full-scale traditional evaluation, and possibly saving valuable resources for researchers. The proposed method uses performance benchmarks obtained from results reported in the literature rather than local simulations. Furthermore, an alternate statistical technique is suggested for comparative performance analysis, since traditional statistical significance tests do not fit the problem well. We highlight the use of the proposed evaluation method in a study that compared a new algorithm against 47 other algorithms across 46 datasets.