Task Scheduling with Makespan Minimization for Distributed Machine Learning Ensembles

Machine Learning problems are becoming increasingly complex, mostly due to the size of datasets. Data are also generated at increasing speed, which requires models to be updated regularly, at a significant computational cost. The project Continuously Evolving Distributed Ensembles proposes the creation of a distributed Machine Learning environment, in which datasets are divided into fixed-size blocks, and stored in a fault-tolerant distributed file system with replication. The base-models of the Ensembles, with a 1:1 relationship with data blocks, are then trained in a distributed manner, according to the principle of data locality. Specifically, the system is able to select which data blocks to use and in which nodes of the cluster, in order to minimize training time. A similar process takes place when making predictions: the best base-models are selected in real-time, according to their predictive performance and to the state of the nodes where they reside. This paper addresses the problem of assigning base model training tasks to cluster nodes, adhering to the principle of data locality. We present an instance generator and three datasets that will provide a means for comparison while studying other solution methods. For testing the system architecture, we solved the datasets with an exact method and the computational results validate, to comply to the project requirements, the need for a more stable and less demanding (in computational resource terms) solution method.