基于代用模型的高性能计算ESM参数调整科学工作流管理框架

IF 2.7 4区 地球科学 Q2 COMPUTER SCIENCE, INTERDISCIPLINARY APPLICATIONS Earth Science Informatics Pub Date : 2024-09-04 DOI:10.1007/s12145-024-01460-x
Liang Hu, Xianwei Wu, Xilong Che
{"title":"基于代用模型的高性能计算ESM参数调整科学工作流管理框架","authors":"Liang Hu, Xianwei Wu, Xilong Che","doi":"10.1007/s12145-024-01460-x","DOIUrl":null,"url":null,"abstract":"<p>In the present era, scientific computation is gradually becoming a primary research method, with an increasing number of researchers engaging in simulation studies on various high-performance computing platforms. Scientific workflows play a crucial role in organizing these complex research tasks effectively. However, poorly managed scientific workflows can lead to wastage of HPC computational resources and fail to alleviate the operational burden on researchers. The parameter optimization of Earth System Models (ESM) poses specific challenges due to its complexity, exacerbating these issues. To address these challenges, we propose a scientific workflow management framework for surrogate-based ESM parameter optimization. This framework consists of four layers: the resource layer, which gathers current resource information; the service layer, which provides various components to ensure the accurate execution of workflows; the management layer, which monitors the execution status of workflows; and the software environment interaction layer, which serves as the interface for data exchange between users and the framework. We monitored a team engaged in tuning CAM parameters before and after adopting the framework, and the results showed significant improvements in operation numbers, task execution time, and storage resource consumption after deploying the framework. This validates that our proposed scientific workflow management framework effectively addresses the challenges in user operations and resource management during surrogate-based ESM optimization processes, demonstrating the potential of our framework.</p>","PeriodicalId":49318,"journal":{"name":"Earth Science Informatics","volume":null,"pages":null},"PeriodicalIF":2.7000,"publicationDate":"2024-09-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":"{\"title\":\"A surrogate model-based ESM parameter tuning scientific workflow management framework for HPC\",\"authors\":\"Liang Hu, Xianwei Wu, Xilong Che\",\"doi\":\"10.1007/s12145-024-01460-x\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"<p>In the present era, scientific computation is gradually becoming a primary research method, with an increasing number of researchers engaging in simulation studies on various high-performance computing platforms. Scientific workflows play a crucial role in organizing these complex research tasks effectively. However, poorly managed scientific workflows can lead to wastage of HPC computational resources and fail to alleviate the operational burden on researchers. The parameter optimization of Earth System Models (ESM) poses specific challenges due to its complexity, exacerbating these issues. To address these challenges, we propose a scientific workflow management framework for surrogate-based ESM parameter optimization. This framework consists of four layers: the resource layer, which gathers current resource information; the service layer, which provides various components to ensure the accurate execution of workflows; the management layer, which monitors the execution status of workflows; and the software environment interaction layer, which serves as the interface for data exchange between users and the framework. We monitored a team engaged in tuning CAM parameters before and after adopting the framework, and the results showed significant improvements in operation numbers, task execution time, and storage resource consumption after deploying the framework. This validates that our proposed scientific workflow management framework effectively addresses the challenges in user operations and resource management during surrogate-based ESM optimization processes, demonstrating the potential of our framework.</p>\",\"PeriodicalId\":49318,\"journal\":{\"name\":\"Earth Science Informatics\",\"volume\":null,\"pages\":null},\"PeriodicalIF\":2.7000,\"publicationDate\":\"2024-09-04\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"Earth Science Informatics\",\"FirstCategoryId\":\"89\",\"ListUrlMain\":\"https://doi.org/10.1007/s12145-024-01460-x\",\"RegionNum\":4,\"RegionCategory\":\"地球科学\",\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"Q2\",\"JCRName\":\"COMPUTER SCIENCE, INTERDISCIPLINARY APPLICATIONS\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"Earth Science Informatics","FirstCategoryId":"89","ListUrlMain":"https://doi.org/10.1007/s12145-024-01460-x","RegionNum":4,"RegionCategory":"地球科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q2","JCRName":"COMPUTER SCIENCE, INTERDISCIPLINARY APPLICATIONS","Score":null,"Total":0}
引用次数: 0

摘要

当今时代,科学计算逐渐成为一种主要的研究方法,越来越多的研究人员在各种高性能计算平台上从事模拟研究。科学工作流在有效组织这些复杂的研究任务方面发挥着至关重要的作用。然而,科学工作流管理不善会导致高性能计算资源的浪费,无法减轻研究人员的操作负担。地球系统模型(ESM)的参数优化因其复杂性而面临特殊挑战,加剧了这些问题。为了应对这些挑战,我们提出了一个基于代理的 ESM 参数优化科学工作流管理框架。该框架由四层组成:资源层,收集当前资源信息;服务层,提供各种组件以确保工作流的准确执行;管理层,监控工作流的执行状态;软件环境交互层,作为用户与框架之间的数据交换接口。在采用该框架前后,我们对参与调整 CAM 参数的团队进行了监测,结果显示,部署该框架后,操作数、任务执行时间和存储资源消耗都有了显著改善。这验证了我们提出的科学工作流管理框架有效地解决了基于代理的 ESM 优化过程中用户操作和资源管理方面的挑战,展示了我们框架的潜力。
本文章由计算机程序翻译,如有差异,请以英文原文为准。

摘要图片

查看原文
分享 分享
微信好友 朋友圈 QQ好友 复制链接
本刊更多论文
A surrogate model-based ESM parameter tuning scientific workflow management framework for HPC

In the present era, scientific computation is gradually becoming a primary research method, with an increasing number of researchers engaging in simulation studies on various high-performance computing platforms. Scientific workflows play a crucial role in organizing these complex research tasks effectively. However, poorly managed scientific workflows can lead to wastage of HPC computational resources and fail to alleviate the operational burden on researchers. The parameter optimization of Earth System Models (ESM) poses specific challenges due to its complexity, exacerbating these issues. To address these challenges, we propose a scientific workflow management framework for surrogate-based ESM parameter optimization. This framework consists of four layers: the resource layer, which gathers current resource information; the service layer, which provides various components to ensure the accurate execution of workflows; the management layer, which monitors the execution status of workflows; and the software environment interaction layer, which serves as the interface for data exchange between users and the framework. We monitored a team engaged in tuning CAM parameters before and after adopting the framework, and the results showed significant improvements in operation numbers, task execution time, and storage resource consumption after deploying the framework. This validates that our proposed scientific workflow management framework effectively addresses the challenges in user operations and resource management during surrogate-based ESM optimization processes, demonstrating the potential of our framework.

求助全文
通过发布文献求助,成功后即可免费获取论文全文。 去求助
来源期刊
Earth Science Informatics
Earth Science Informatics COMPUTER SCIENCE, INTERDISCIPLINARY APPLICATIONS-GEOSCIENCES, MULTIDISCIPLINARY
CiteScore
4.60
自引率
3.60%
发文量
157
审稿时长
4.3 months
期刊介绍: The Earth Science Informatics [ESIN] journal aims at rapid publication of high-quality, current, cutting-edge, and provocative scientific work in the area of Earth Science Informatics as it relates to Earth systems science and space science. This includes articles on the application of formal and computational methods, computational Earth science, spatial and temporal analyses, and all aspects of computer applications to the acquisition, storage, processing, interchange, and visualization of data and information about the materials, properties, processes, features, and phenomena that occur at all scales and locations in the Earth system’s five components (atmosphere, hydrosphere, geosphere, biosphere, cryosphere) and in space (see "About this journal" for more detail). The quarterly journal publishes research, methodology, and software articles, as well as editorials, comments, and book and software reviews. Review articles of relevant findings, topics, and methodologies are also considered.
期刊最新文献
Estimation of the elastic modulus of basaltic rocks using machine learning methods Feature-adaptive FPN with multiscale context integration for underwater object detection Autoregressive modelling of tropospheric radio refractivity over selected locations in tropical Nigeria using artificial neural network Time series land subsidence monitoring and prediction based on SBAS-InSAR and GeoTemporal transformer model Drought index time series forecasting via three-in-one machine learning concept for the Euphrates basin
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
现在去查看 取消
×
提示
确定
0
微信
客服QQ
Book学术公众号 扫码关注我们
反馈
×
意见反馈
请填写您的意见或建议
请填写您的手机或邮箱
已复制链接
已复制链接
快去分享给好友吧!
我知道了
×
扫码分享
扫码分享
Book学术官方微信
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术
文献互助 智能选刊 最新文献 互助须知 联系我们:info@booksci.cn
Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。
Copyright © 2023 Book学术 All rights reserved.
ghs 京公网安备 11010802042870号 京ICP备2023020795号-1