A Physical and Virtual Compute Cluster Resource Load Balancing Approach to Data-Parallel Scientific Workflow Scheduling

2011 IEEE World Congress on Services Pub Date : 2011-07-04 DOI:10.1109/SERVICES.2011.50

Jianwu Wang, P. Korambath, I. Altintas

引用次数: 13

Abstract

To execute workflows on a compute cluster resource, workflow engines can work with cluster resource manager software to distribute jobs into compute nodes on the cluster. We discuss how to interact with traditional Oracle Grid Engine and recent Hadoop cluster resource managers using a dataflow-based scheduling approach to balance compute resource load for data-parallel workflow execution. Our experiments show that: 1) The presented approach can balance computational resource load well by interacting with the resource managers and provides good execution performance on both physical and virtual clusters, 2) Oracle Grid Engine outperforms Hadoop for CPU-intensive applications on small-scale clusters.

查看原文

微信好友朋友圈 QQ好友复制链接

本刊更多论文

数据并行科学工作流调度的物理和虚拟计算集群资源负载均衡方法

要在计算集群资源上执行工作流，工作流引擎可以与集群资源管理器软件一起工作，将作业分发到集群上的计算节点。我们讨论了如何使用基于数据流的调度方法与传统的Oracle Grid Engine和最近的Hadoop集群资源管理器进行交互，以平衡数据并行工作流执行的计算资源负载。实验结果表明:1)该方法通过与资源管理器的交互，可以很好地平衡计算资源负载，在物理集群和虚拟集群上都提供了良好的执行性能;2)Oracle Grid Engine在小规模集群上对于cpu密集型应用的性能优于Hadoop。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文去求助

来源期刊

2011 IEEE World Congress on Services

自引率

0.00%

发文量

期刊最新文献

Reputation-Based Web Service Selection for Composition SSC4Cloud Tooling: An Integrated Environment for the Development of Business Processes with Security Requirements in the Cloud Simplifying Web Service Discovery & Validating Service Composition A Survey of Cloud Storage Facilities Externalizing the Autopoietic Part of Software to Achieve Self-Adaptability