Pengyan Shen, Kai Guo, Mingzhong Xiao, Quanqing Xu
Constructing an overlay storage system based on multiple personal cloud storages is a desirable technique and novel idea for cloud storages. Existing designs provide the basic functions with some customized features. Unfortunately, some important issues have always been ignored including privacy protection, QoS and cipher-text search. In this paper, we present Spy, our design for an anonymous storage overlay network on multiple personal cloud storage, supporting a flexible QoS awareness and cipher-text search. We reform the original Tor protocol by extending the command set and adding a tail part to the Tor cell, which makes it possible for coordination among proxy servers and still keeps the anonymity. Based on which, we proposed a flexible user-defined QoS policy and employed a Dynamic Searchable Symmetric Encryption (DSSE) scheme to support secure cipher-text search. Extensive security analysis prove the security on privacy preserving and experiments show how different QoS policy work according to different security requirements.
{"title":"Spy: A QoS-Aware Anonymous Multi-Cloud Storage System Supporting DSSE","authors":"Pengyan Shen, Kai Guo, Mingzhong Xiao, Quanqing Xu","doi":"10.1109/CCGrid.2015.88","DOIUrl":"https://doi.org/10.1109/CCGrid.2015.88","url":null,"abstract":"Constructing an overlay storage system based on multiple personal cloud storages is a desirable technique and novel idea for cloud storages. Existing designs provide the basic functions with some customized features. Unfortunately, some important issues have always been ignored including privacy protection, QoS and cipher-text search. In this paper, we present Spy, our design for an anonymous storage overlay network on multiple personal cloud storage, supporting a flexible QoS awareness and cipher-text search. We reform the original Tor protocol by extending the command set and adding a tail part to the Tor cell, which makes it possible for coordination among proxy servers and still keeps the anonymity. Based on which, we proposed a flexible user-defined QoS policy and employed a Dynamic Searchable Symmetric Encryption (DSSE) scheme to support secure cipher-text search. Extensive security analysis prove the security on privacy preserving and experiments show how different QoS policy work according to different security requirements.","PeriodicalId":6664,"journal":{"name":"2015 15th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing","volume":"17 1","pages":"951-960"},"PeriodicalIF":0.0,"publicationDate":"2015-05-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"75318429","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Christopher Smowton, Crispin J. Miller, W. Xing, Andoena Balla, D. Antoniades, G. Pallis, M. Dikaiakos
With the rapidly growing demand for DNA analysis, the need for storing and processing large-scale genome data has presented significant challenges. This paper describes how the Genome Analysis Toolkit (GATK) can be deployed to an elastic cloud, and defines policy to drive elastic scaling of the application. We extensively analyse the GATK to expose opportunities for resource elasticity, demonstrate that it can be practically deployed at scale in a cloud environment, and demonstrate that applying elastic scaling improves the performance to cost tradeoff achieved in a simulated environment.
{"title":"Analysing Cancer Genomics in the Elastic Cloud","authors":"Christopher Smowton, Crispin J. Miller, W. Xing, Andoena Balla, D. Antoniades, G. Pallis, M. Dikaiakos","doi":"10.1109/CCGRID.2015.176","DOIUrl":"https://doi.org/10.1109/CCGRID.2015.176","url":null,"abstract":"With the rapidly growing demand for DNA analysis, the need for storing and processing large-scale genome data has presented significant challenges. This paper describes how the Genome Analysis Toolkit (GATK) can be deployed to an elastic cloud, and defines policy to drive elastic scaling of the application. We extensively analyse the GATK to expose opportunities for resource elasticity, demonstrate that it can be practically deployed at scale in a cloud environment, and demonstrate that applying elastic scaling improves the performance to cost tradeoff achieved in a simulated environment.","PeriodicalId":6664,"journal":{"name":"2015 15th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing","volume":"80 1","pages":"835-844"},"PeriodicalIF":0.0,"publicationDate":"2015-05-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"75905599","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Heterogeneous platforms integrating different types of processing units (such as multi-core CPUs and GPUs) are in high demand in high performance computing. Existing studies have shown that using heterogeneous platforms can improve application performance and hardware utilization. However, systematic methods to design, implement, and map applications to efficiently use heterogeneous computing resources are only very few. The goal of my PhD research is therefore to study such heterogeneous systems and propose systematic methods to allow many (classes of) applications to efficiently use them. After 3.5 years of PhD study, my contributions are (1) a thorough evaluation of a suitable programming model for heterogeneous computing, (2) a workload partitioning framework to accelerate parallel applications on heterogeneous platforms, (3) a modelling-based prediction method to determine the optimal workload partitioning, (4) a systematic approach to decide the best mapping between the application and the platform by choosing the best performing hardware configuration (Only-CPU, Only-GPU, or CPU+GPU with the workload partitioning). In the near future, I plan to apply my approach to large-scale applications and platforms to expand its usability and applicability.
{"title":"Improving Application Performance by Efficiently Utilizing Heterogeneous Many-core Platforms","authors":"Jie Shen, A. Varbanescu, H. Sips","doi":"10.1109/CCGrid.2015.44","DOIUrl":"https://doi.org/10.1109/CCGrid.2015.44","url":null,"abstract":"Heterogeneous platforms integrating different types of processing units (such as multi-core CPUs and GPUs) are in high demand in high performance computing. Existing studies have shown that using heterogeneous platforms can improve application performance and hardware utilization. However, systematic methods to design, implement, and map applications to efficiently use heterogeneous computing resources are only very few. The goal of my PhD research is therefore to study such heterogeneous systems and propose systematic methods to allow many (classes of) applications to efficiently use them. After 3.5 years of PhD study, my contributions are (1) a thorough evaluation of a suitable programming model for heterogeneous computing, (2) a workload partitioning framework to accelerate parallel applications on heterogeneous platforms, (3) a modelling-based prediction method to determine the optimal workload partitioning, (4) a systematic approach to decide the best mapping between the application and the platform by choosing the best performing hardware configuration (Only-CPU, Only-GPU, or CPU+GPU with the workload partitioning). In the near future, I plan to apply my approach to large-scale applications and platforms to expand its usability and applicability.","PeriodicalId":6664,"journal":{"name":"2015 15th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing","volume":"23 1","pages":"709-712"},"PeriodicalIF":0.0,"publicationDate":"2015-05-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"80099177","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Emerging big data applications comprise rich multi-faceted workflows with both compute-intensive and data-intensive tasks, and intricate communication patterns. While MapReduce is an effective model for data-intensive tasks, the MPI programming model may be better suited for extracting high-performance for compute-intensive tasks. Researchers have recognized this need to employ specialized models for different phases of a workflow, e.g., performing computations using MPI followed by visualizations using MapReduce. However, extant multi-cluster approaches are inefficient as they entail data movement across clusters and porting across data formats. Consequently, there is a crucial need for disparate programming models to co-exist on the same set of resources. In this paper, we address the above issue by designing GERBIL, a framework for transparently co-hosting unmodified MPI applications alongside MapReduce applications on the same cluster. GERBIL exploits YARN as the model agnostic resource negotiator, and provides an easy-to-use interface to the users. GERBIL bridges the fundamental mismatch between YARN and MPI by designing an MPI-aware resource allocation mechanism. We also support five different optimizations: minimizing job wait time, achieving inter-process locality, achieving desired cluster utilization, minimizing network traffic, and minimizing job execution time, all in a multi-tenant environment. Our evaluation shows that GERBIL enables MPI executions with performance comparable to a native MPI setup, and improve compute-intensive applications performance by up to 133% when compared to the corresponding MapReduce-based versions.
{"title":"GERBIL: MPI+YARN","authors":"Luna Xu, Min Li, Ali Raza Butt","doi":"10.1109/CCGrid.2015.137","DOIUrl":"https://doi.org/10.1109/CCGrid.2015.137","url":null,"abstract":"Emerging big data applications comprise rich multi-faceted workflows with both compute-intensive and data-intensive tasks, and intricate communication patterns. While MapReduce is an effective model for data-intensive tasks, the MPI programming model may be better suited for extracting high-performance for compute-intensive tasks. Researchers have recognized this need to employ specialized models for different phases of a workflow, e.g., performing computations using MPI followed by visualizations using MapReduce. However, extant multi-cluster approaches are inefficient as they entail data movement across clusters and porting across data formats. Consequently, there is a crucial need for disparate programming models to co-exist on the same set of resources. In this paper, we address the above issue by designing GERBIL, a framework for transparently co-hosting unmodified MPI applications alongside MapReduce applications on the same cluster. GERBIL exploits YARN as the model agnostic resource negotiator, and provides an easy-to-use interface to the users. GERBIL bridges the fundamental mismatch between YARN and MPI by designing an MPI-aware resource allocation mechanism. We also support five different optimizations: minimizing job wait time, achieving inter-process locality, achieving desired cluster utilization, minimizing network traffic, and minimizing job execution time, all in a multi-tenant environment. Our evaluation shows that GERBIL enables MPI executions with performance comparable to a native MPI setup, and improve compute-intensive applications performance by up to 133% when compared to the corresponding MapReduce-based versions.","PeriodicalId":6664,"journal":{"name":"2015 15th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing","volume":"36 1","pages":"627-636"},"PeriodicalIF":0.0,"publicationDate":"2015-05-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"77363377","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Hang Zhuang, Kun Lu, Changlong Li, Mingming Sun, Hang Chen, Xuehai Zhou
With the development of cloud computing and internet, e-Commerce, e-Business and corporate world revenue are increasing with high rate. These areas require scalable and consistent databases. NoSQL databases such as HBase has been proven to scalability and well performance on cloud computing platforms. However, the inevitable special data with few increment and frequent access leads to hotspot data and unbalanced accessing distribution between data storage servers. Due to their properties, these data often cannot be stored in multiple tables. Some storage nodes become the bottleneck of the distributed storage system, therefore, it becomes difficult to improve the performance by increasing the number of nodes which severely limits the scalability of the storage system. In order to make the performance of the cluster increases with the size of the cluster simultaneously, we devise a new distributed database storage framework to solve those issues mentioned above by changing the storage and read-write mode of the hotspot data. This structure guarantees that the hotspot data will not aggregate in the same storage node, as it guarantees that the data is not too hot in a single storage node. We implement the scalable database based on Apache HBase, which achieve almost double performance of throughput considering heavy read-write pressure situation only with double reading substites. Besides, heavy load node owing to hotspot data will no longer present in the new distributed database.
{"title":"Design of a More Scalable Database System","authors":"Hang Zhuang, Kun Lu, Changlong Li, Mingming Sun, Hang Chen, Xuehai Zhou","doi":"10.1109/CCGrid.2015.70","DOIUrl":"https://doi.org/10.1109/CCGrid.2015.70","url":null,"abstract":"With the development of cloud computing and internet, e-Commerce, e-Business and corporate world revenue are increasing with high rate. These areas require scalable and consistent databases. NoSQL databases such as HBase has been proven to scalability and well performance on cloud computing platforms. However, the inevitable special data with few increment and frequent access leads to hotspot data and unbalanced accessing distribution between data storage servers. Due to their properties, these data often cannot be stored in multiple tables. Some storage nodes become the bottleneck of the distributed storage system, therefore, it becomes difficult to improve the performance by increasing the number of nodes which severely limits the scalability of the storage system. In order to make the performance of the cluster increases with the size of the cluster simultaneously, we devise a new distributed database storage framework to solve those issues mentioned above by changing the storage and read-write mode of the hotspot data. This structure guarantees that the hotspot data will not aggregate in the same storage node, as it guarantees that the data is not too hot in a single storage node. We implement the scalable database based on Apache HBase, which achieve almost double performance of throughput considering heavy read-write pressure situation only with double reading substites. Besides, heavy load node owing to hotspot data will no longer present in the new distributed database.","PeriodicalId":6664,"journal":{"name":"2015 15th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing","volume":"18 1","pages":"1213-1216"},"PeriodicalIF":0.0,"publicationDate":"2015-05-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"74338266","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Graph processing is increasingly used in knowledge economies and in science, in advanced marketing, social networking, bioinformatics, etc. A number of graph-processing systems, including the GPU-enabled Medusa and Totem, have been developed recently. Understanding their performance is key to system selection, tuning, and improvement. Previous performance evaluation studies have been conducted for CPU-based graph-processing systems, such as Graph and GraphX. Unlike them, the performance of GPU-enabled systems is still not thoroughly evaluated and compared. To address this gap, we propose an empirical method for evaluating GPU-enabled graph-processing systems, which includes new performance metrics and a selection of new datasets and algorithms. By selecting 9 diverse graphs and 3 typical graph-processing algorithms, we conduct a comparative performance study of 3 GPU-enabled systems, Medusa, Totem, and MapGraph. We present the first comprehensive evaluation of GPU-enabled systems with results giving insight into raw processing power, performance breakdown into core components, scalability, and the impact on performance of system-specific optimization techniques and of the GPU generation. We present and discuss many findings that would benefit users and developers interested in GPU acceleration for graph processing.
{"title":"An Empirical Performance Evaluation of GPU-Enabled Graph-Processing Systems","authors":"Yong Guo, A. Varbanescu, A. Iosup, D. Epema","doi":"10.1109/CCGrid.2015.20","DOIUrl":"https://doi.org/10.1109/CCGrid.2015.20","url":null,"abstract":"Graph processing is increasingly used in knowledge economies and in science, in advanced marketing, social networking, bioinformatics, etc. A number of graph-processing systems, including the GPU-enabled Medusa and Totem, have been developed recently. Understanding their performance is key to system selection, tuning, and improvement. Previous performance evaluation studies have been conducted for CPU-based graph-processing systems, such as Graph and GraphX. Unlike them, the performance of GPU-enabled systems is still not thoroughly evaluated and compared. To address this gap, we propose an empirical method for evaluating GPU-enabled graph-processing systems, which includes new performance metrics and a selection of new datasets and algorithms. By selecting 9 diverse graphs and 3 typical graph-processing algorithms, we conduct a comparative performance study of 3 GPU-enabled systems, Medusa, Totem, and MapGraph. We present the first comprehensive evaluation of GPU-enabled systems with results giving insight into raw processing power, performance breakdown into core components, scalability, and the impact on performance of system-specific optimization techniques and of the GPU generation. We present and discuss many findings that would benefit users and developers interested in GPU acceleration for graph processing.","PeriodicalId":6664,"journal":{"name":"2015 15th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing","volume":"75 1","pages":"423-432"},"PeriodicalIF":0.0,"publicationDate":"2015-05-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"86352028","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
CUDA is a technology developed by NVIDIA which provides a parallel computing platform and programming model for NVIDIA GPUs and compatible ones. It takes benefit from the enormous parallel processing power of GPUs in order to accelerate a wide range of applications, thus reducing their execution time. rCUDA (remote CUDA) is a middleware which grants applications concurrent access to CUDA-compatible devices installed in other nodes of the cluster in a transparent way so that applications are not aware of being accessing a remote device. In this paper we present a demo which shows, in real time, the overhead introduced by rCUDA in comparison to CUDA when running image filtering applications. The approach followed in this work is to develop a graphical demo which contains both an appealing design and technical contents.
{"title":"On the Design of a Demo for Exhibiting rCUDA","authors":"C. Reaño, Ferran Perez, F. Silla","doi":"10.1109/CCGrid.2015.53","DOIUrl":"https://doi.org/10.1109/CCGrid.2015.53","url":null,"abstract":"CUDA is a technology developed by NVIDIA which provides a parallel computing platform and programming model for NVIDIA GPUs and compatible ones. It takes benefit from the enormous parallel processing power of GPUs in order to accelerate a wide range of applications, thus reducing their execution time. rCUDA (remote CUDA) is a middleware which grants applications concurrent access to CUDA-compatible devices installed in other nodes of the cluster in a transparent way so that applications are not aware of being accessing a remote device. In this paper we present a demo which shows, in real time, the overhead introduced by rCUDA in comparison to CUDA when running image filtering applications. The approach followed in this work is to develop a graphical demo which contains both an appealing design and technical contents.","PeriodicalId":6664,"journal":{"name":"2015 15th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing","volume":"35 1","pages":"1169-1172"},"PeriodicalIF":0.0,"publicationDate":"2015-05-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"85086134","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
The device people use to capture multimedia has changed over the years with the rise of smart phones. Smart phones are readily available, easy to use, and capture multimedia with high quality. While consumers capture all of this media, the storage requirements are not changing significantly. Therefore, people look towards cloud storage solutions. The typical consumer stores files within a single provider. They want a solution that is quick to access, reliable, and secure. Using multiple providers can reduce cost and improve overall performance. We present a middleware framework called Distributed Indexed Storage in the Cloud (DISC) to improve all aspects a user expects in a cloud provider. The process of uploading and downloading is essentially transparent to the user. The upload and download performance happens simultaneously by distributing a subset of the file across multiple cloud providers that it deems fit based on policies. Reliability is another important feature of DISC. To improve reliability, we propose a solution that replicates the same subset of the file across different providers. This is beneficial when one provider is unresponsive, the data can be pulled from another provider with the same subset. Security has great importance when dealing with consumers data. We inherently gain security when improving reliability. Since the file is distributed using subsets, not one provider has the full file. In our experiment, performance improvements are observed when delivering and retrieving files compared to the standard approach. The results are promising, saving upwards of eight seconds in processing time. With the expansion of more cloud providers, the results are expected to improve.
{"title":"Augmenting Performance For Distributed Cloud Storage","authors":"Matthew B. Hancock, Carlos A. Varela","doi":"10.1109/CCGrid.2015.124","DOIUrl":"https://doi.org/10.1109/CCGrid.2015.124","url":null,"abstract":"The device people use to capture multimedia has changed over the years with the rise of smart phones. Smart phones are readily available, easy to use, and capture multimedia with high quality. While consumers capture all of this media, the storage requirements are not changing significantly. Therefore, people look towards cloud storage solutions. The typical consumer stores files within a single provider. They want a solution that is quick to access, reliable, and secure. Using multiple providers can reduce cost and improve overall performance. We present a middleware framework called Distributed Indexed Storage in the Cloud (DISC) to improve all aspects a user expects in a cloud provider. The process of uploading and downloading is essentially transparent to the user. The upload and download performance happens simultaneously by distributing a subset of the file across multiple cloud providers that it deems fit based on policies. Reliability is another important feature of DISC. To improve reliability, we propose a solution that replicates the same subset of the file across different providers. This is beneficial when one provider is unresponsive, the data can be pulled from another provider with the same subset. Security has great importance when dealing with consumers data. We inherently gain security when improving reliability. Since the file is distributed using subsets, not one provider has the full file. In our experiment, performance improvements are observed when delivering and retrieving files compared to the standard approach. The results are promising, saving upwards of eight seconds in processing time. With the expansion of more cloud providers, the results are expected to improve.","PeriodicalId":6664,"journal":{"name":"2015 15th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing","volume":"23 1","pages":"1189-1192"},"PeriodicalIF":0.0,"publicationDate":"2015-05-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"91026285","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Alexandru Uta, Andreea Sandu, S. Costache, T. Kielmann
Data-intensive scientific workflows are composed of many tasks that exhibit data precedence constraints leading to communication schemes expressed by means of intermediate files. In such scenarios, the storage layer is often a bottleneck, limiting overall application scalability, due to large volumes of data being generated during runtime at high I/O rates. To alleviate the storage pressure, applications take advantage of in-memory runtime distributed file systems that act as a fast, distributed cache, which greatly enhances I/O performance.In this paper, we present scalability results for MemFS, a distributed in-memory runtime file system. MemFS takes an opposite approach to data locality, by scattering all data among the nodes, leading to well balanced storage and network traffic, and thus making the system both highly per formant and scalable. Our results show that MemFS is platform independent, performing equally well on both private clusters and commercial clouds. On such platforms, running on up to 1024 cores, MemFS shows excellent horizontal scalability (using more nodes), while the vertical scalability (using more cores per node) is only limited by the network b and with. Further more, for this challenge we show how MemFS is able to scale elastically, at runtime, based on the application storage demands. In our experiments, we have successfully used up to 1TB memory when running a large instance of the Montage workflow.
{"title":"Scalable In-Memory Computing","authors":"Alexandru Uta, Andreea Sandu, S. Costache, T. Kielmann","doi":"10.1109/CCGrid.2015.106","DOIUrl":"https://doi.org/10.1109/CCGrid.2015.106","url":null,"abstract":"Data-intensive scientific workflows are composed of many tasks that exhibit data precedence constraints leading to communication schemes expressed by means of intermediate files. In such scenarios, the storage layer is often a bottleneck, limiting overall application scalability, due to large volumes of data being generated during runtime at high I/O rates. To alleviate the storage pressure, applications take advantage of in-memory runtime distributed file systems that act as a fast, distributed cache, which greatly enhances I/O performance.In this paper, we present scalability results for MemFS, a distributed in-memory runtime file system. MemFS takes an opposite approach to data locality, by scattering all data among the nodes, leading to well balanced storage and network traffic, and thus making the system both highly per formant and scalable. Our results show that MemFS is platform independent, performing equally well on both private clusters and commercial clouds. On such platforms, running on up to 1024 cores, MemFS shows excellent horizontal scalability (using more nodes), while the vertical scalability (using more cores per node) is only limited by the network b and with. Further more, for this challenge we show how MemFS is able to scale elastically, at runtime, based on the application storage demands. In our experiments, we have successfully used up to 1TB memory when running a large instance of the Montage workflow.","PeriodicalId":6664,"journal":{"name":"2015 15th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing","volume":"10 1","pages":"805-810"},"PeriodicalIF":0.0,"publicationDate":"2015-05-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"90155076","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
In this paper, we introduce a majority quorum system dedicated to p-m-n general threshold schemes where p, n and m are respectively the minimal number of chunks that provide some information (but not necessarily all) on the original data, the total number of nodes in which the chunks of an object are stored and the minimal number of nodes needed to retrieve the original data using this protocol. In other words, less than p chunks reveal absolutely no information about the original data and less than m chunks can't reconstruct the original data. The p-m-n general threshold schemes optimize the usage of storage resources by reducing the total size of data to write and ensure fault-tolerance up to (n - m) nodes failure. With such a data distribution, a specific value of m can be set to have a good trade off between resources utilization and fault-tolerance. The only drawback of such schemes is the lack of any consistency protocol. If fact, consistency protocols like classical majority quorum are based on full replication. To successfully read or write a data using the majority quorum protocol, an absolute majority of replicas must be read / written correctly. This condition ensures that any read and write operations will contain at least one common replica, which guarantees their consistency. However, when a threshold scheme is used, an adaptation is needed. In fact, classical majority quorum protocol can no longer ensure that m chunks will have the latest version when [n/2] + 1 <; m ≤ n. In this paper, we introduce a new majority quorum protocol dedicated to general threshold schemes. As for the classical majority quorum protocol, the complexity of the quorum size of our protocol is O(n) but the utilization of storage resources is greatly optimized.
{"title":"Majority Quorum Protocol Dedicated to General Threshold Schemes","authors":"T. J. R. Relaza, J. Jorda, A. Mzoughi","doi":"10.1109/CCGrid.2015.76","DOIUrl":"https://doi.org/10.1109/CCGrid.2015.76","url":null,"abstract":"In this paper, we introduce a majority quorum system dedicated to p-m-n general threshold schemes where p, n and m are respectively the minimal number of chunks that provide some information (but not necessarily all) on the original data, the total number of nodes in which the chunks of an object are stored and the minimal number of nodes needed to retrieve the original data using this protocol. In other words, less than p chunks reveal absolutely no information about the original data and less than m chunks can't reconstruct the original data. The p-m-n general threshold schemes optimize the usage of storage resources by reducing the total size of data to write and ensure fault-tolerance up to (n - m) nodes failure. With such a data distribution, a specific value of m can be set to have a good trade off between resources utilization and fault-tolerance. The only drawback of such schemes is the lack of any consistency protocol. If fact, consistency protocols like classical majority quorum are based on full replication. To successfully read or write a data using the majority quorum protocol, an absolute majority of replicas must be read / written correctly. This condition ensures that any read and write operations will contain at least one common replica, which guarantees their consistency. However, when a threshold scheme is used, an adaptation is needed. In fact, classical majority quorum protocol can no longer ensure that m chunks will have the latest version when [n/2] + 1 <; m ≤ n. In this paper, we introduce a new majority quorum protocol dedicated to general threshold schemes. As for the classical majority quorum protocol, the complexity of the quorum size of our protocol is O(n) but the utilization of storage resources is greatly optimized.","PeriodicalId":6664,"journal":{"name":"2015 15th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing","volume":"30 1","pages":"785-788"},"PeriodicalIF":0.0,"publicationDate":"2015-05-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"84730930","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：