Hypothesis: Net benefit as an objective function during development of machine learning algorithms for medical applications

Abstract

Net benefit is the most widely used metric for evaluating the clinical utility of medical prediction models. The approach applies decision analytic theory to weight true and false positives depending on the relative consequences of different decision outcomes. It is plausible that there are at least some machine learning scenarios where optimization of the objective function during model development will not optimize net benefit during model evaluation. We therefore hypothesize that optimizing net benefit during model development will in some cases ultimately lead to higher clinical utility than optimizing for mean square error or some other unweighted loss function. There is some preliminary evidence that this does indeed occur. We accordingly recommend further methodologic research to determine the use cases where net benefit should be the objective function during model development.