Software for Prediction of Task Start Moment in Computer Cluster by Statistical Analysis of Jobs Queue History

Abstract

Predicting the start time of a task without placing the task in a task queue is very difficult in modern data centers. The article proposes to improve the previously created forecasting tool through the use of more modern software interfaces to the SchedMD queuing system (Slurm), as well as the use of various mathematical forecasting methods to predict the moment of task launch. To compare and evaluate machine learning models, we selected a SWF file with task execution history data on the University of Luxembourg computing cluster with 59 715 tasks. The following methods were investigated: linear regression with L2 regularization, support vector machine, random forest, LightGBM, CatBoost, LightGBM with parameter optimization, CatBoost with parameter optimization. To check the correctness of the prediction, a test bench was built based on the Slurm simulator (SUNY Center for Computational Research at the University at Buffalo, USA), which works based on executing tasks and organizing a queue of saved logs.