Validation in the age of machine learning: A framework for describing validation with examples in transcranial magnetic stimulation and deep brain stimulation

Medical information processing is a staple of modern medicine with its increasing focus on the collection of numeric medical data such as questionnaires, biophysiological signals, and medical images. Although these modalities have long existed and guided medical practice, the movement towards using algorithms to transform, curate, summarise, and otherwise interact with this data is relatively new. Novel algorithms now form the interface between clinical users and data, extracting information that would otherwise be inaccessible or cumbersome. Recently, machine learning has expanded the capacities of these algorithms, using a priori acquired (and often annotated) datasets to learn a complex computational task. Validation of these techniques is inherently important for determining their safety and efficacy in a particular clinical context. However, methodological considerations such as the definition of reference data and validation procedures can obscure validation issues such as inaccurate reporting, a lack of standardisation, and a variety of biases. The purpose of this paper is to develop a framework for understanding medical information processing algorithms with a focus on validation that is adapted for machine learning approaches as well as traditional ones. This framework is instantiated in two example literature reviews which serve as the starting point for a discussion on how validation can be improved cognisant of machine learning.