Scheduling Constrained Cloud Workflow Tasks via Evolutionary Multitasking Optimization With Adaptive Knowledge Transfer

Abstract

Cloud workflow scheduling (CWS) is critical for meeting user's high performance expectations in large-scale data processing and computing applications. CWS is known to be NP-hard and needs advanced scheduling techniques. Evolutionary algorithm and heuristic-based search techniques have gained massive popularity in addressing CWS, yet they either suffer from expensive computational cost or heavily rely on domain-specific experiences, which limit their practical applications. Bearing this in mind, we develop a novel evolutionary multi-task optimization framework to tackle a group of constrained CWS tasks simultaneously with the aid of adaptive cross-task problem-solving knowledge transfer. In particular, two collaborative knowledge exchange strategies, namely, constraint-free archive strategy and cross-task evolution strategy, are devised to extract useful building blocks from foreign tasks to boost the search efficiency. Further, to leverage the cooperative effects of both strategies, we develop an adaptive switching mechanism such that appropriate knowledge transfer strategies are learned automatically according to the population evolution status. Extensive experiments are conducted on real-world applications under various conditions, the comparison results show that our proposal delivers higher quality schedules than the state-of-the-art competitors in most cases.