A surrogate model-based ESM parameter tuning scientific workflow management framework for HPC

Abstract

In the present era, scientific computation is gradually becoming a primary research method, with an increasing number of researchers engaging in simulation studies on various high-performance computing platforms. Scientific workflows play a crucial role in organizing these complex research tasks effectively. However, poorly managed scientific workflows can lead to wastage of HPC computational resources and fail to alleviate the operational burden on researchers. The parameter optimization of Earth System Models (ESM) poses specific challenges due to its complexity, exacerbating these issues. To address these challenges, we propose a scientific workflow management framework for surrogate-based ESM parameter optimization. This framework consists of four layers: the resource layer, which gathers current resource information; the service layer, which provides various components to ensure the accurate execution of workflows; the management layer, which monitors the execution status of workflows; and the software environment interaction layer, which serves as the interface for data exchange between users and the framework. We monitored a team engaged in tuning CAM parameters before and after adopting the framework, and the results showed significant improvements in operation numbers, task execution time, and storage resource consumption after deploying the framework. This validates that our proposed scientific workflow management framework effectively addresses the challenges in user operations and resource management during surrogate-based ESM optimization processes, demonstrating the potential of our framework.