A Kubernetes-based scheme for efficient resource allocation in containerized workflow

Abstract

In the cloud-native era, Kubernetes-based workflow engines simplify the execution of containerized workflows. However, these engines face challenges in dynamic environments with continuous workflow requests and unpredictable resource demand peaks. The traditional resource allocation approach, which relies merely on current workflow load data, also lacks flexibility and foresight, often leading to resource over-allocation or scarcity. To tackle these issues, we present a containerized workflow resource allocation (CWRA) scheme designed specifically for Kubernetes workflow engines. CWRA predicts future workflow tasks during the current task pod’s lifecycle and employs a dynamic resource scaling strategy to manage high concurrency scenarios effectively. This scheme includes resource discovery and allocation algorithm, which are essential components of our containerized workflow engine (CWE). Our experimental results, across various workflow arrival patterns, indicate significant improvements when compared to the Argo workflow engine. CWRA achieves a reduction in total workflow duration by 0.9% to 11.4%, decreases average workflow duration by a maximum of 21.5%, and increases CPU and memory utilization by 2.07% to 16.95%.