Determining resource allocations (memory and time) for submitted jobs in High Performance Computing (HPC) systems is a challenging process even for computer scientists. HPC users are highly encouraged to overestimate resource allocation for their submitted jobs, so their jobs will not be killed due to insufficient resources. Overestimating resource allocations occurs because of the wide variety of HPC applications and environment configuration options, and the lack of knowledge of the complex structure of HPC systems. This causes a waste of HPC resources, a decreased utilization of HPC systems, and increased waiting and turnaround time for submitted jobs. In this paper, we introduce our first ever implemented fully-offline, fully-automated, stand-alone, and open-source Machine Learning (ML) tool to help users predict memory and time requirements for their submitted jobs on the cluster. Our tool involves implementing six ML discriminative models from the scikit-learn and Microsoft LightGBM applied on the historical data (sacct data) from Simple Linux Utility for Resource Management (Slurm). We have tested our tool using historical data (saact data) using HPC resources of Kansas State University (Beocat), which covers the years from January 2019 - March 2021, and contains around 17.6 million jobs. Our results show that our tool achieves high predictive accuracy R2 (0.72 using LightGBM for predicting the memory and 0.74 using Random Forest for predicting the time), helps dramatically reduce computational average waiting-time and turnaround time for the submitted jobs, and increases utilization of the HPC resources. Hence, our tool decreases the power consumption of the HPC resources.
在高性能计算(HPC)系统中，为提交的作业确定资源分配(内存和时间)是一个具有挑战性的过程，即使对计算机科学家也是如此。强烈建议HPC用户高估其提交作业的资源分配，这样他们的作业就不会因为资源不足而被终止。由于HPC应用程序和环境配置选项的多样性，以及缺乏对HPC系统复杂结构的了解，会出现对资源分配的高估。这会导致HPC资源的浪费，HPC系统的利用率降低，以及提交作业的等待和周转时间增加。在本文中，我们介绍了我们有史以来第一个实现的完全离线、全自动、独立和开源的机器学习(ML)工具，以帮助用户预测他们在集群上提交的作业的内存和时间需求。我们的工具包括实现来自scikit-learn和Microsoft LightGBM的6个ML判别模型，这些模型应用于来自Simple Linux Utility for Resource Management (Slurm)的历史数据(sact数据)。我们使用堪萨斯州立大学(Beocat)的HPC资源使用历史数据(saact数据)测试了我们的工具，这些数据涵盖了2019年1月至2021年3月的年份，包含了大约1760万个工作岗位。我们的结果表明，我们的工具达到了很高的预测精度r2(使用LightGBM预测内存为0.72，使用Random Forest预测时间为0.74)，有助于显着减少提交作业的计算平均等待时间和周转时间，并提高HPC资源的利用率。因此，我们的工具降低了HPC资源的功耗。
{"title":"AMPRO-HPCC: A Machine-Learning Tool for Predicting Resources on Slurm HPC Clusters.","authors":"Mohammed Tanash, Daniel Andresen, William Hsu","doi":"","DOIUrl":"","url":null,"abstract":"<p><p>Determining resource allocations (memory and time) for submitted jobs in High Performance Computing (HPC) systems is a challenging process even for computer scientists. HPC users are highly encouraged to overestimate resource allocation for their submitted jobs, so their jobs will not be killed due to insufficient resources. Overestimating resource allocations occurs because of the wide variety of HPC applications and environment configuration options, and the lack of knowledge of the complex structure of HPC systems. This causes a waste of HPC resources, a decreased utilization of HPC systems, and increased waiting and turnaround time for submitted jobs. In this paper, we introduce our first ever implemented fully-offline, fully-automated, stand-alone, and open-source Machine Learning (ML) tool to help users predict memory and time requirements for their submitted jobs on the cluster. Our tool involves implementing six ML discriminative models from the scikit-learn and Microsoft LightGBM applied on the historical data (sacct data) from Simple Linux Utility for Resource Management (Slurm). We have tested our tool using historical data (saact data) using HPC resources of Kansas State University (Beocat), which covers the years from January 2019 - March 2021, and contains around 17.6 million jobs. Our results show that our tool achieves high predictive accuracy <i>R</i> <sup>2</sup> (0.72 using LightGBM for predicting the memory and 0.74 using Random Forest for predicting the time), helps dramatically reduce computational average waiting-time and turnaround time for the submitted jobs, and increases utilization of the HPC resources. Hence, our tool decreases the power consumption of the HPC resources.</p>","PeriodicalId":72112,"journal":{"name":"ADVCOMP ... the ... International Conference on Advanced Engineering Computing and Applications in Sciences","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2021-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9906793/pdf/nihms-1831252.pdf","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"10760547","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Research data management is becoming increasingly complex as the amount of data, metadata and code increases. Often, researchers must obtain multidisciplinary skills to acquire, transfer, share, and compute large datasets. In this paper we present the results of an investigation into providing a familiar web-based experience for researchers to manage their data and code, leveraging popular, well-funded tools and services. We show how researchers can save time and avoid mistakes, and we provide a detailed discussion of our system architecture and implementation, and summarize the new capabilities, and time savings which can be achieved.
{"title":"Facilitating large data management in research contexts.","authors":"Daniel Andresen, Gerrick Teague","doi":"","DOIUrl":"","url":null,"abstract":"<p><p>Research data management is becoming increasingly complex as the amount of data, metadata and code increases. Often, researchers must obtain multidisciplinary skills to acquire, transfer, share, and compute large datasets. In this paper we present the results of an investigation into providing a familiar web-based experience for researchers to manage their data and code, leveraging popular, well-funded tools and services. We show how researchers can save time and avoid mistakes, and we provide a detailed discussion of our system architecture and implementation, and summarize the new capabilities, and time savings which can be achieved.</p>","PeriodicalId":72112,"journal":{"name":"ADVCOMP ... the ... International Conference on Advanced Engineering Computing and Applications in Sciences","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2021-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9446462/pdf/nihms-1831850.pdf","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"33448466","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2020-01-01DOI: 10.1109/ACOMP50827.2020.00005
M. Marchese, Lam-Son Lê, Bob Dao, M. Toulouse, N. Thoai
{"title":"Message from the Program Chairs and Industry Panel Chairs","authors":"M. Marchese, Lam-Son Lê, Bob Dao, M. Toulouse, N. Thoai","doi":"10.1109/ACOMP50827.2020.00005","DOIUrl":"https://doi.org/10.1109/ACOMP50827.2020.00005","url":null,"abstract":"","PeriodicalId":72112,"journal":{"name":"ADVCOMP ... the ... International Conference on Advanced Engineering Computing and Applications in Sciences","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2020-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"74407752","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2019-11-01DOI: 10.1109/ACOMP.2019.00025
P. N. Huu, Q. Minh, Hieu Nguyen Trong
This paper proposes an effective key-generation scheme applying to data encryption standard (DES) algorithm for wireless sensor networks (WSNs). In the scheme, data encryption is divided into several tasks for multiple nodes along a path from a source node to the base station. We perform simulations to compare distribution and centralization models. The results show that the distributed model obtains more balances in energy consumption compared to the centralization model. The proposed key management method also improves the security level of data by increasing the number of keys with a simple algorithm in WSNs.
{"title":"Low-Complexity Encryption Algorithm Considering Energy Balance on Wireless Sensor Networks","authors":"P. N. Huu, Q. Minh, Hieu Nguyen Trong","doi":"10.1109/ACOMP.2019.00025","DOIUrl":"https://doi.org/10.1109/ACOMP.2019.00025","url":null,"abstract":"This paper proposes an effective key-generation scheme applying to data encryption standard (DES) algorithm for wireless sensor networks (WSNs). In the scheme, data encryption is divided into several tasks for multiple nodes along a path from a source node to the base station. We perform simulations to compare distribution and centralization models. The results show that the distributed model obtains more balances in energy consumption compared to the centralization model. The proposed key management method also improves the security level of data by increasing the number of keys with a simple algorithm in WSNs.","PeriodicalId":72112,"journal":{"name":"ADVCOMP ... the ... International Conference on Advanced Engineering Computing and Applications in Sciences","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2019-11-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"74012966","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2019-11-01DOI: 10.1109/ACOMP.2019.00012
A. An, P. Diem, L. Lan, Tran Van Toi, Lam Quoc Huy Le Nguyen Binh
In recent years, the traceability of product origins is strongly concerned, particularly for food products as they directly influence human health. Therefore, there have been some efforts to develop product origins tracking systems. In this paper, we propose an approach to building a supply chain management system based on the blockchain technology for agriculture product origins tracking. The supply chain model is borrowed from Walmart's and it is implemented based on the Ethereum framework using the PoA (Proof of Authority) consensus algorithm. Our experiment shows that the proposed system not only fulfills the requirements of a product origins tracking but also takes the advantages of the blockchain technology such as the immutability and security of data, the low cost in making the transactions, and so on.
{"title":"Building a Product Origins Tracking System Based on Blockchain and PoA Consensus Protocol","authors":"A. An, P. Diem, L. Lan, Tran Van Toi, Lam Quoc Huy Le Nguyen Binh","doi":"10.1109/ACOMP.2019.00012","DOIUrl":"https://doi.org/10.1109/ACOMP.2019.00012","url":null,"abstract":"In recent years, the traceability of product origins is strongly concerned, particularly for food products as they directly influence human health. Therefore, there have been some efforts to develop product origins tracking systems. In this paper, we propose an approach to building a supply chain management system based on the blockchain technology for agriculture product origins tracking. The supply chain model is borrowed from Walmart's and it is implemented based on the Ethereum framework using the PoA (Proof of Authority) consensus algorithm. Our experiment shows that the proposed system not only fulfills the requirements of a product origins tracking but also takes the advantages of the blockchain technology such as the immutability and security of data, the low cost in making the transactions, and so on.","PeriodicalId":72112,"journal":{"name":"ADVCOMP ... the ... International Conference on Advanced Engineering Computing and Applications in Sciences","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2019-11-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"75423339","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
扫码关注我们
求助内容：
应助结果提醒方式：
京公网安备 11010802042870号