Optimizing Resource Estimation for Scientific Workflows in HPC Environments: A Layered-Bucket Heuristic Approach

Abstract

As computational simulations become complex and the amount of processed data grows, executing scientific workflows in High-Performance Computing (HPC) environments is increasingly essential. However, accurately estimating the required computational resources for such executions presents a significant challenge, requiring a thorough examination of the workflow structure and the characteristics of the computational environment. This manuscript introduces the GraspCC-LB heuristic, based on the Greedy Randomized Adaptive Search Procedure (GRASP), for estimating the necessary resources for executing scientific workflows in HPC environments. Unlike existing methods, GraspCC-LB incorporates the layered structure of workflows into its estimation process. The proposed approach was evaluated using real traces of workflows from the fields of bioinformatics and astronomy. The resource estimations produced by GraspCC-LB were compared against the actual resource usage in a real-world HPC environment to evaluate its effectiveness. The results demonstrate the effectiveness of GraspCC-LB as a robust approach for resource optimization in the context of large-scale scientific workflows that require HPC capabilities.