Knowledge Transfer Enabled Diverse Task Scheduling for Individualized Requirements in Industrial Cloud Platform

Abstract

Nowadays, application providers often prefer to execute their workflows on heterogeneous distributed computing resources deployed on cloud infrastructure to achieve a high level of resilience and cost saving. Optimally scheduling workflow on computing resources is a well-known combinatorial optimization problem, where a trend of using evolutionary algorithm (EA) is emerging rapidly. However, conventional EA optimizes only one problem in a single run and suffers from a high computational burden. In practical scenario, cloud platform needs to handle massive amounts of scheduling requests from users, scheduling different workflows simultaneously is highly challenging. Bearing this in mind, we put forward a novel knowledge transfer enabled EA to schedule diverse workflows in tandem, where domain knowledge of scheduling one workflow is extracted to enhance the scheduling efficiency of other related workflows. In our design, the knowledge source selection and the intensity of performing knowledge transfer are adapted in a synergistic way. Furthermore, search operator is enhanced by exploiting both historical experience and heuristic information. Experimental results on real-life workflows and extensive synthetic applications demonstrate the competitiveness of our approach, in comparison to state-of-the-art contenders. Note to Practitioners—Workflow scheduling is an important requirement for users in cloud computing, whose intractability increases exponentially when the size of problem grows, posing stiff challenges to heuristic methods. Using EAs to tackle workflow scheduling has received increasing attention recently. Suppose workflow scheduling is treated as a optimization task, cloud platform typically needs to handle versatile tasks from numerous users. However, traditional EA optimizes only one task in a single run and unable to handle multiple tasks at the same time. To address this issue, we introduce a novel multi-task solver to resolve different tasks jointly via online learning and exploitation of problem-solving experiences across tasks. The results demonstrate that our proposal significantly outperforms the state-of-the-art peers. It is expected to facilitate the practical efficacy of industrial cloud system which faces multiple workflow scheduling tasks submitted from enormous users.