Organizations and individuals nowadays face increasing daily operations closely rely on a huge amount of private data which is outsourced to a centralized server. Secure and efficient data processing and mining on such outsourced private data becomes a primary concern for users, especially with the push of cloud computing which has both resource and compute scalability. Among the building blocks of secure data mining algorithms, secure scalar product is used to calculate the sum of the products of the corresponding values of two vectors. Existing privacy preserving methods assume data is stored at the user side, and users follow a protocol to perform privacy preserving scalar product. However, such methods are not applicable as data now is outsourced to a centralized server in its encrypted form. To solve this problem, in this paper, we design a novel Protocol for Outsourced Scalar Product (POSP) that performs collaborative operations between server and users to produce the scalar product result without violating each user's data privacy. We proved that POSP can return the correct result and is secure. We also analysed that POSP has linear complexity in terms of space, computation, and communication with respect to the vector length.
{"title":"Encrypted Scalar Product Protocol for Outsourced Data Mining","authors":"Fang Liu, W. Ng, Wei Zhang","doi":"10.1109/CLOUD.2014.53","DOIUrl":"https://doi.org/10.1109/CLOUD.2014.53","url":null,"abstract":"Organizations and individuals nowadays face increasing daily operations closely rely on a huge amount of private data which is outsourced to a centralized server. Secure and efficient data processing and mining on such outsourced private data becomes a primary concern for users, especially with the push of cloud computing which has both resource and compute scalability. Among the building blocks of secure data mining algorithms, secure scalar product is used to calculate the sum of the products of the corresponding values of two vectors. Existing privacy preserving methods assume data is stored at the user side, and users follow a protocol to perform privacy preserving scalar product. However, such methods are not applicable as data now is outsourced to a centralized server in its encrypted form. To solve this problem, in this paper, we design a novel Protocol for Outsourced Scalar Product (POSP) that performs collaborative operations between server and users to produce the scalar product result without violating each user's data privacy. We proved that POSP can return the correct result and is secure. We also analysed that POSP has linear complexity in terms of space, computation, and communication with respect to the vector length.","PeriodicalId":288542,"journal":{"name":"2014 IEEE 7th International Conference on Cloud Computing","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2014-06-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"127885162","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Network virtualization enables the multi-tenancy concept and paves the way towards more advancements and innovation in the underlying infrastructure. With network virtualization, allocating resources to Virtual Networks (VNs) that represent tenants' requests emerges as a challenging problem. This problem is commonly known as the Virtual Network Embedding (VNE) problem, and its NP-Hard nature has drawn a lot of attention from the research community. A common feature in the existing work is that the type of communication in the VN requests was never characterized, assuming that they exhibit unicast communication only. In this paper, we motivate the importance of characterizing the type of communication in VN requests. We present a formal definition of the VNE problem for VNs with multicast communication. To the best of our knowledge, the multicast VNE problem has not been addressed in the frame of cloud computing, where the location of all the virtual machines in a given multicast VN is unknown. We propose a novel 3-steps heuristic to solve the multicast VNE problem with end-delay and delay variation constraints. Our numerical results prove the efficiency of our suggested approach over multiple metrics and against numerous embedding heuristics.
{"title":"Multicast Virtual Network Embedding in Cloud Data Centers with Delay Constraints","authors":"Sara Ayoubi, K. Shaban, C. Assi","doi":"10.1109/CLOUD.2014.71","DOIUrl":"https://doi.org/10.1109/CLOUD.2014.71","url":null,"abstract":"Network virtualization enables the multi-tenancy concept and paves the way towards more advancements and innovation in the underlying infrastructure. With network virtualization, allocating resources to Virtual Networks (VNs) that represent tenants' requests emerges as a challenging problem. This problem is commonly known as the Virtual Network Embedding (VNE) problem, and its NP-Hard nature has drawn a lot of attention from the research community. A common feature in the existing work is that the type of communication in the VN requests was never characterized, assuming that they exhibit unicast communication only. In this paper, we motivate the importance of characterizing the type of communication in VN requests. We present a formal definition of the VNE problem for VNs with multicast communication. To the best of our knowledge, the multicast VNE problem has not been addressed in the frame of cloud computing, where the location of all the virtual machines in a given multicast VN is unknown. We propose a novel 3-steps heuristic to solve the multicast VNE problem with end-delay and delay variation constraints. Our numerical results prove the efficiency of our suggested approach over multiple metrics and against numerous embedding heuristics.","PeriodicalId":288542,"journal":{"name":"2014 IEEE 7th International Conference on Cloud Computing","volume":"6 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2014-06-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"126563924","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Nowadays, virtualization is used as the sole mechanism to isolate different users on Cloud platforms. In this paper, we show that, due to improper virtualization of micro-architectural components, data leak and modification can occur on public Clouds. Furthermore, using the same vector, it is possible to induce performance interferences, i.e. noisy neighbors. Using this approach, a VM can steal resources from, and slow down, concurrent VMs. To counter this, we propose placement heuristics that take into account isolation requirements, thus allowing a user to specify the level of isolation he accepts, and with whom. We modify 3 classical heuristics to take into account these requirements. In addition, we propose 4 new heuristics that take into account the hierarchy of Cloud platforms and isolation requirements. Finally, we evaluate these heuristics and compare them with the modified classical ones. We show that our heuristics perform at least as well as the classical ones, while scaling better and being faster by a few orders of magnitude.
{"title":"Improving Users' Isolation in IaaS: Virtual Machine Placement with Security Constraints","authors":"E. Caron, Jonathan Rouzaud-Cornabas","doi":"10.1109/CLOUD.2014.19","DOIUrl":"https://doi.org/10.1109/CLOUD.2014.19","url":null,"abstract":"Nowadays, virtualization is used as the sole mechanism to isolate different users on Cloud platforms. In this paper, we show that, due to improper virtualization of micro-architectural components, data leak and modification can occur on public Clouds. Furthermore, using the same vector, it is possible to induce performance interferences, i.e. noisy neighbors. Using this approach, a VM can steal resources from, and slow down, concurrent VMs. To counter this, we propose placement heuristics that take into account isolation requirements, thus allowing a user to specify the level of isolation he accepts, and with whom. We modify 3 classical heuristics to take into account these requirements. In addition, we propose 4 new heuristics that take into account the hierarchy of Cloud platforms and isolation requirements. Finally, we evaluate these heuristics and compare them with the modified classical ones. We show that our heuristics perform at least as well as the classical ones, while scaling better and being faster by a few orders of magnitude.","PeriodicalId":288542,"journal":{"name":"2014 IEEE 7th International Conference on Cloud Computing","volume":"372 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2014-06-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"115905146","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Maintaining high reliability and device utilization in a cloud computing system (CCS) is crucial to any cloud service provider who will face high penalties and lose revenues if they fail to be good at both. This study proposes that allowing device partial failure in a CCS for graceful service degrading would help to obtain higher system reliability and device utilization without purchasing extra resource for the system. A model is created to represent such a multi-state system composed of multi-state devices. The system model is evaluated with Non-sequential Monte Carlo Simulation (MCS) on its reliability and device utilization. The preliminary results positively suggest that introducing and adding device multi-state increases the CCS reliability against device failures during simulation. Also, for the less reliable devices, like HDD, the results recommended a higher multi-state to compensate for their vulnerability and negative effect on system performance. A utilization index along all device dimensions is proposed in this research for a wise decision about maintaining a well-balanced and high utilized system at a lower cost.
{"title":"Reliability and Utilization Evaluation of a Cloud Computing System Allowing Partial Failures","authors":"Congyingzi Zhang, Robert C. Green, Mansoor Alam","doi":"10.1109/CLOUD.2014.131","DOIUrl":"https://doi.org/10.1109/CLOUD.2014.131","url":null,"abstract":"Maintaining high reliability and device utilization in a cloud computing system (CCS) is crucial to any cloud service provider who will face high penalties and lose revenues if they fail to be good at both. This study proposes that allowing device partial failure in a CCS for graceful service degrading would help to obtain higher system reliability and device utilization without purchasing extra resource for the system. A model is created to represent such a multi-state system composed of multi-state devices. The system model is evaluated with Non-sequential Monte Carlo Simulation (MCS) on its reliability and device utilization. The preliminary results positively suggest that introducing and adding device multi-state increases the CCS reliability against device failures during simulation. Also, for the less reliable devices, like HDD, the results recommended a higher multi-state to compensate for their vulnerability and negative effect on system performance. A utilization index along all device dimensions is proposed in this research for a wise decision about maintaining a well-balanced and high utilized system at a lower cost.","PeriodicalId":288542,"journal":{"name":"2014 IEEE 7th International Conference on Cloud Computing","volume":"34 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2014-06-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"131549612","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
M. Salehi, T. Caldwell, Alejandro Fernandez, Emmanuel Mickiewicz, Eric Rozier, S. Zonouz, David Redberg
Capabilities for trustworthy cloud-based computing and data storage require usable, secure and efficient solutions which allow clients to remotely store and process their data in the cloud. In this paper, we present RESeED, a tool which provides user-transparent and cloud-agnostic search over encrypted data using regular expressions without requiring cloud providers to make changes to their existing infrastructure. When a client asks RESeED to upload a new file in the cloud, RESeED analyzes the file's content and updates novel data structures accordingly, encrypting and transferring the new data to the cloud. RESeED provides regular expression search over this encrypted data by translating queries on-the-fly to finite automata and analyzes efficient and secure representations of the data before asking the cloud to download the encrypted files. We evaulate a working prototype of RESeED experimentally (currently publicly available) and show the scalability and correctness of our approach using real-world data sets from arXiv.org and the IETF. We show absolute accuracy for RESeED, with very low (6%) overhead, and high performability, even beating grep for some benchmarks.
{"title":"RESeED: Regular Expression Search over Encrypted Data in the Cloud","authors":"M. Salehi, T. Caldwell, Alejandro Fernandez, Emmanuel Mickiewicz, Eric Rozier, S. Zonouz, David Redberg","doi":"10.1109/CLOUD.2014.95","DOIUrl":"https://doi.org/10.1109/CLOUD.2014.95","url":null,"abstract":"Capabilities for trustworthy cloud-based computing and data storage require usable, secure and efficient solutions which allow clients to remotely store and process their data in the cloud. In this paper, we present RESeED, a tool which provides user-transparent and cloud-agnostic search over encrypted data using regular expressions without requiring cloud providers to make changes to their existing infrastructure. When a client asks RESeED to upload a new file in the cloud, RESeED analyzes the file's content and updates novel data structures accordingly, encrypting and transferring the new data to the cloud. RESeED provides regular expression search over this encrypted data by translating queries on-the-fly to finite automata and analyzes efficient and secure representations of the data before asking the cloud to download the encrypted files. We evaulate a working prototype of RESeED experimentally (currently publicly available) and show the scalability and correctness of our approach using real-world data sets from arXiv.org and the IETF. We show absolute accuracy for RESeED, with very low (6%) overhead, and high performability, even beating grep for some benchmarks.","PeriodicalId":288542,"journal":{"name":"2014 IEEE 7th International Conference on Cloud Computing","volume":"77 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2014-06-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"131259386","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Florian Haupt, F. Leymann, Alexander Nowak, S. Wagner
The Topology and Orchestration Specification for Cloud Applications (TOSCA) enables the description, provisioning, and management of complex cloud applications in a portable way. TOSCA, therefore, provides a comprehensive yet complex set of mechanisms that may hinder users from unleashing its power due to misusing or neglecting parts of those mechanisms. TOSCA has just been standardized and, although it seems to be highly adopted in industry, there is a lack of systematic research of its features and capabilities. In this work we discuss the design of basic building blocks for cloud applications, called node types, and show how they can benefit from a deep integration with TOSCA. We developed a generic architecture for the realization of TOSCA node types, show an implementation of this architecture and validate it based on a sample cloud application. Our work gives an insight into the capabilities of TOSCA with respect to enable the creation of portable cloud services based on a set of composable building blocks.
{"title":"Lego4TOSCA: Composable Building Blocks for Cloud Applications","authors":"Florian Haupt, F. Leymann, Alexander Nowak, S. Wagner","doi":"10.1109/CLOUD.2014.31","DOIUrl":"https://doi.org/10.1109/CLOUD.2014.31","url":null,"abstract":"The Topology and Orchestration Specification for Cloud Applications (TOSCA) enables the description, provisioning, and management of complex cloud applications in a portable way. TOSCA, therefore, provides a comprehensive yet complex set of mechanisms that may hinder users from unleashing its power due to misusing or neglecting parts of those mechanisms. TOSCA has just been standardized and, although it seems to be highly adopted in industry, there is a lack of systematic research of its features and capabilities. In this work we discuss the design of basic building blocks for cloud applications, called node types, and show how they can benefit from a deep integration with TOSCA. We developed a generic architecture for the realization of TOSCA node types, show an implementation of this architecture and validate it based on a sample cloud application. Our work gives an insight into the capabilities of TOSCA with respect to enable the creation of portable cloud services based on a set of composable building blocks.","PeriodicalId":288542,"journal":{"name":"2014 IEEE 7th International Conference on Cloud Computing","volume":"34 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2014-06-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"128891595","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Nowadays, scientists are embracing big data techniques for exploring significant discoveries from large volumes of scientific data quickly. Properly partitioning workloads is essential for fully exploiting the benefit of parallelism, but is difficult for applications whose computations change iteratively. Computation skews are inevitable when executing block-oriented iterative scientific applications in MapReduce systems. This paper proposes iPart, an autonomic workload partitioning system for taming computation skews of block-oriented iterative scientific applications in MapReduce systems. iPart introduces a workload control loop into the conventional execution of MapReduce jobs. Workload estimates in terms of execution time are collected in the reduce phase and fed back to the partition phase to update partitioning plans. Computation skews are detected and addressed by adapting partitioning to computation changes iteratively. Two adaptive partitioning methods based on the binary partitioning method are presented. Experimental evaluations with two simulated applications and the synthetic and real-world data prove that iPart responds to computation changes and adapts partitioning quickly and accurately.
{"title":"Taming Computation Skews of Block-Oriented Iterative Scientific Applications in MapReduce Systems","authors":"Xin Yang, Min Li, Ze Yu, Xiaolin Li","doi":"10.1109/CLOUD.2014.33","DOIUrl":"https://doi.org/10.1109/CLOUD.2014.33","url":null,"abstract":"Nowadays, scientists are embracing big data techniques for exploring significant discoveries from large volumes of scientific data quickly. Properly partitioning workloads is essential for fully exploiting the benefit of parallelism, but is difficult for applications whose computations change iteratively. Computation skews are inevitable when executing block-oriented iterative scientific applications in MapReduce systems. This paper proposes iPart, an autonomic workload partitioning system for taming computation skews of block-oriented iterative scientific applications in MapReduce systems. iPart introduces a workload control loop into the conventional execution of MapReduce jobs. Workload estimates in terms of execution time are collected in the reduce phase and fed back to the partition phase to update partitioning plans. Computation skews are detected and addressed by adapting partitioning to computation changes iteratively. Two adaptive partitioning methods based on the binary partitioning method are presented. Experimental evaluations with two simulated applications and the synthetic and real-world data prove that iPart responds to computation changes and adapts partitioning quickly and accurately.","PeriodicalId":288542,"journal":{"name":"2014 IEEE 7th International Conference on Cloud Computing","volume":"3 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2014-06-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"124140858","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
With the increased adoption of cloud computing, new challenges have emerged related to the cost-effective management of cloud resources. The proliferation of resource properties and pricing plans has made the selection, procurement and management of cloud resources a time-consuming and complex task, which stands to benefit from automation. This contribution focuses on the procurement decision of reserved contracts in the context of Infrastructure-as-a-Service (IaaS) providers such as Amazon EC2. Such reserved contracts complement pay-by-the-hour pricing models, and offer a significant reduction in price (up to 70%) for a particular period in return for an upfront payment. Thus, customers can reduce costs by predicting and analyzing their future needs in terms of the number and type of server instances. We present an algorithm that uses load prediction with automated time series forecasting based on a Double-seasonal Holt-Winters model, in order to make cost-efficient purchasing decisions among a wide range of contract types while taking into account an organization's current contract portfolio. We analyze its cost effectiveness through simulation of real-world web traffic traces. Our analysis investigates the impact of different prediction techniques on cost compared to a clairvoyant predictor and compares the algorithm's performance with a stationary contract renewal approach. Our results show that the algorithm is able to significantly reduce IaaS resource costs through automated reserved contract procurement. Moreover, the algorithm's computational cost makes it applicable to large-scale real-world settings.
{"title":"Optimizing IaaS Reserved Contract Procurement Using Load Prediction","authors":"R. Y. V. Bossche, K. Vanmechelen, J. Broeckhove","doi":"10.1109/CLOUD.2014.22","DOIUrl":"https://doi.org/10.1109/CLOUD.2014.22","url":null,"abstract":"With the increased adoption of cloud computing, new challenges have emerged related to the cost-effective management of cloud resources. The proliferation of resource properties and pricing plans has made the selection, procurement and management of cloud resources a time-consuming and complex task, which stands to benefit from automation. This contribution focuses on the procurement decision of reserved contracts in the context of Infrastructure-as-a-Service (IaaS) providers such as Amazon EC2. Such reserved contracts complement pay-by-the-hour pricing models, and offer a significant reduction in price (up to 70%) for a particular period in return for an upfront payment. Thus, customers can reduce costs by predicting and analyzing their future needs in terms of the number and type of server instances. We present an algorithm that uses load prediction with automated time series forecasting based on a Double-seasonal Holt-Winters model, in order to make cost-efficient purchasing decisions among a wide range of contract types while taking into account an organization's current contract portfolio. We analyze its cost effectiveness through simulation of real-world web traffic traces. Our analysis investigates the impact of different prediction techniques on cost compared to a clairvoyant predictor and compares the algorithm's performance with a stationary contract renewal approach. Our results show that the algorithm is able to significantly reduce IaaS resource costs through automated reserved contract procurement. Moreover, the algorithm's computational cost makes it applicable to large-scale real-world settings.","PeriodicalId":288542,"journal":{"name":"2014 IEEE 7th International Conference on Cloud Computing","volume":"150 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2014-06-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"123441140","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Cloud computing requires elasticity management for dynamically allocating and releasing resources. Even though the adoption of cloud services has been growing, there is little knowledge available for guiding users when they need to manage elasticity. This paper analyzes elasticity in private and hybrid clouds, using a university lab, PlanetLab and Amazon EC2. Results show that the choice of metrics and thresholds plays a key role in meeting performance levels and controlling costs and that cloudburst can be effectively used for a hybrid cloud but the choice of the type of virtual machine in the provider has a significant impact.
{"title":"Elasticity Management in Private and Hybrid Clouds","authors":"Rhodney Simões, C. Kamienski","doi":"10.1109/CLOUD.2014.110","DOIUrl":"https://doi.org/10.1109/CLOUD.2014.110","url":null,"abstract":"Cloud computing requires elasticity management for dynamically allocating and releasing resources. Even though the adoption of cloud services has been growing, there is little knowledge available for guiding users when they need to manage elasticity. This paper analyzes elasticity in private and hybrid clouds, using a university lab, PlanetLab and Amazon EC2. Results show that the choice of metrics and thresholds plays a key role in meeting performance levels and controlling costs and that cloudburst can be effectively used for a hybrid cloud but the choice of the type of virtual machine in the provider has a significant impact.","PeriodicalId":288542,"journal":{"name":"2014 IEEE 7th International Conference on Cloud Computing","volume":"9 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2014-06-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"121925580","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Yi Yao, Jiayin Wang, B. Sheng, Jason H. Lin, N. Mi
The MapReduce framework has become the de facto scheme for scalable semi-structured and un-structured data processing in recent years. The Hadoop ecosystem has evolved into its second generation, Hadoop YARN, which adopts fine-grained resource management schemes for job scheduling. One of the primary performance concerns in YARN is how to minimize the total completion length, i.e., makespan, of a set of MapReduce jobs. However, the precedence constraint or fairness constraint in current widely used scheduling policies in YARN, such as FIFO and Fair, can both lead to inefficient resource allocation in the Hadoop YARN cluster. They also omit the dependency between tasks which is crucial for the efficiency of resource utilization. We thus propose a new YARN scheduler, named HaSTE, which can effectively reduce the makespan of MapReduce jobs in YARN by leveraging the information of requested resources, resource capacities, and dependency between tasks. We implemented HaSTE as a pluggable scheduler in the most recent version of Hadoop YARN, and evaluated it with classic MapReduce benchmarks. The experimental results demonstrate that our YARN scheduler effectively reduces the makespans and improves resource utilization compare to the current scheduling policies.
{"title":"HaSTE: Hadoop YARN Scheduling Based on Task-Dependency and Resource-Demand","authors":"Yi Yao, Jiayin Wang, B. Sheng, Jason H. Lin, N. Mi","doi":"10.1109/CLOUD.2014.34","DOIUrl":"https://doi.org/10.1109/CLOUD.2014.34","url":null,"abstract":"The MapReduce framework has become the de facto scheme for scalable semi-structured and un-structured data processing in recent years. The Hadoop ecosystem has evolved into its second generation, Hadoop YARN, which adopts fine-grained resource management schemes for job scheduling. One of the primary performance concerns in YARN is how to minimize the total completion length, i.e., makespan, of a set of MapReduce jobs. However, the precedence constraint or fairness constraint in current widely used scheduling policies in YARN, such as FIFO and Fair, can both lead to inefficient resource allocation in the Hadoop YARN cluster. They also omit the dependency between tasks which is crucial for the efficiency of resource utilization. We thus propose a new YARN scheduler, named HaSTE, which can effectively reduce the makespan of MapReduce jobs in YARN by leveraging the information of requested resources, resource capacities, and dependency between tasks. We implemented HaSTE as a pluggable scheduler in the most recent version of Hadoop YARN, and evaluated it with classic MapReduce benchmarks. The experimental results demonstrate that our YARN scheduler effectively reduces the makespans and improves resource utilization compare to the current scheduling policies.","PeriodicalId":288542,"journal":{"name":"2014 IEEE 7th International Conference on Cloud Computing","volume":"28 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2014-06-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"124808480","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：