A Data-aware MultiWorkflow Scheduler for Clusters on WorkflowSim

César Acevedo, P. Hernández, Antonio Espinosa, Víctor Méndez
{"title":"A Data-aware MultiWorkflow Scheduler for Clusters on WorkflowSim","authors":"César Acevedo, P. Hernández, Antonio Espinosa, Víctor Méndez","doi":"10.5220/0006303500790086","DOIUrl":null,"url":null,"abstract":"Most scientific workflows are defined as Direct Acyclic Graphs. Despite DAGs are very expressive to reflect dependencies relationships, current approaches are not aware of the storage physiognomy in terms of performance and capacity. Provide information about temporal storage allocation on data intensive applications helps to avoid performance issues. Nevertheless, we need to evaluate several combinations of data file locations and application scheduling. Simulation is one of the most popular evaluation methods in scientific workflow execution to develop new storage-aware scheduling techniques or improve existing ones, to test scalability and repetitiveness. This paper presents a multiworkflow store-aware scheduler policy as an extension of WorkflowSim, enabling its combination with other WorkflowSim scheduling policies and the possibility of evaluating a wide range of storage and file allocation possibilities. This paper also presents a proof of concept of a real world implementation of a storage-aware scheduler to validate the accuracy of the WorkflowSim extension and the scalability of our scheduler technique. The evaluation on several environments shows promising results up to 69% of makespan improvement on simulated large scale clusters with an error of the WorflowSim extension between 0,9% and 3% comparing with the real infrastructure implementation.","PeriodicalId":414016,"journal":{"name":"International Conference on Complex Information Systems","volume":"124 1","pages":"0"},"PeriodicalIF":0.0000,"publicationDate":"1900-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"1","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"International Conference on Complex Information Systems","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.5220/0006303500790086","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 1

Abstract

Most scientific workflows are defined as Direct Acyclic Graphs. Despite DAGs are very expressive to reflect dependencies relationships, current approaches are not aware of the storage physiognomy in terms of performance and capacity. Provide information about temporal storage allocation on data intensive applications helps to avoid performance issues. Nevertheless, we need to evaluate several combinations of data file locations and application scheduling. Simulation is one of the most popular evaluation methods in scientific workflow execution to develop new storage-aware scheduling techniques or improve existing ones, to test scalability and repetitiveness. This paper presents a multiworkflow store-aware scheduler policy as an extension of WorkflowSim, enabling its combination with other WorkflowSim scheduling policies and the possibility of evaluating a wide range of storage and file allocation possibilities. This paper also presents a proof of concept of a real world implementation of a storage-aware scheduler to validate the accuracy of the WorkflowSim extension and the scalability of our scheduler technique. The evaluation on several environments shows promising results up to 69% of makespan improvement on simulated large scale clusters with an error of the WorflowSim extension between 0,9% and 3% comparing with the real infrastructure implementation.
查看原文
分享 分享
微信好友 朋友圈 QQ好友 复制链接
本刊更多论文
WorkflowSim上集群的数据感知多工作流调度程序
大多数科学工作流被定义为直接无环图。尽管dag在反映依赖关系方面非常有表现力,但当前的方法并不了解性能和容量方面的存储外观。提供有关数据密集型应用程序的临时存储分配的信息有助于避免性能问题。然而,我们需要评估数据文件位置和应用程序调度的几种组合。仿真是科学工作流执行中最流行的评估方法之一,用于开发新的存储感知调度技术或改进现有调度技术,测试可扩展性和重复性。本文提出了一个多工作流存储感知调度策略作为WorkflowSim的扩展,使其能够与其他WorkflowSim调度策略相结合,并有可能评估大范围的存储和文件分配可能性。本文还提供了一个存储感知调度器的现实世界实现的概念证明,以验证WorkflowSim扩展的准确性和我们的调度器技术的可扩展性。在几个环境下的评估显示了有希望的结果,在模拟的大规模集群上,与真实的基础设施实现相比,WorflowSim扩展的最大完工时间提高了69%,误差在0.9%到3%之间。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 去求助
来源期刊
自引率
0.00%
发文量
0
期刊最新文献
Moving Other Way: Exploring Word Mover Distance Extensions Artificial Neural Networks Jamming on the Beat FLOPTICS: A Novel Automated Gating Technique for Flow Cytometry Data It Means More if It Sounds Good: Yet Another Hypotheses Concerning the Evolution of Polysemous Words An Empirical Research on the Investment Strategy of Stock Market based on Deep Reinforcement Learning model
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
现在去查看 取消
×
提示
确定
0
微信
客服QQ
Book学术公众号 扫码关注我们
反馈
×
意见反馈
请填写您的意见或建议
请填写您的手机或邮箱
已复制链接
已复制链接
快去分享给好友吧!
我知道了
×
扫码分享
扫码分享
Book学术官方微信
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术
文献互助 智能选刊 最新文献 互助须知 联系我们:info@booksci.cn
Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。
Copyright © 2023 Book学术 All rights reserved.
ghs 京公网安备 11010802042870号 京ICP备2023020795号-1