Pub Date : 2018-03-21DOI: 10.1109/PDP2018.2018.00045
Amogh Katti, D. Lilja
This paper proposes a memory efficient failure detection and consensus algorithm, for fail-stop type process failures, based on epidemic protocols. It is suitable for extreme scale systems with reliable networks (no message loss) and high failure frequency. Communication time dominates the execution time at scale. The redundant failure detections and non-uniform information dissemination speed of epidemic algorithms make approximate epidemic-based consensus detection a useful way to trade communication overhead for accuracy. An approximate technique to the consensus detection is also proposed in this paper for faster consensus detection. Results show that the algorithm detects consensus correctly on failed processes with logarithmic scalability. The algorithm is tolerant to process failures both before and during the execution and the number of failures (occurring both before and during execution) have virtually no effect on the consensus detection time at scale. Comparison with similar deterministic consensus detection technique shows that the algorithm detects consensus at the same time with high probability. Further, benefits of the proposed approximate technique increase as system size increases. Compared to the non-approximate version, for a system size of 218 processes, the communication saved is 34% with accuracy loss of the order of 10^-4 in consensus detection.
{"title":"Efficient and Fast Approximate Consensus with Epidemic Failure Detection at Extreme Scale","authors":"Amogh Katti, D. Lilja","doi":"10.1109/PDP2018.2018.00045","DOIUrl":"https://doi.org/10.1109/PDP2018.2018.00045","url":null,"abstract":"This paper proposes a memory efficient failure detection and consensus algorithm, for fail-stop type process failures, based on epidemic protocols. It is suitable for extreme scale systems with reliable networks (no message loss) and high failure frequency. Communication time dominates the execution time at scale. The redundant failure detections and non-uniform information dissemination speed of epidemic algorithms make approximate epidemic-based consensus detection a useful way to trade communication overhead for accuracy. An approximate technique to the consensus detection is also proposed in this paper for faster consensus detection. Results show that the algorithm detects consensus correctly on failed processes with logarithmic scalability. The algorithm is tolerant to process failures both before and during the execution and the number of failures (occurring both before and during execution) have virtually no effect on the consensus detection time at scale. Comparison with similar deterministic consensus detection technique shows that the algorithm detects consensus at the same time with high probability. Further, benefits of the proposed approximate technique increase as system size increases. Compared to the non-approximate version, for a system size of 218 processes, the communication saved is 34% with accuracy loss of the order of 10^-4 in consensus detection.","PeriodicalId":333367,"journal":{"name":"2018 26th Euromicro International Conference on Parallel, Distributed and Network-based Processing (PDP)","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2018-03-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"114509094","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2018-03-21DOI: 10.1109/PDP2018.2018.00022
Amin Majd, A. Ashraf, E. Troubitsyna, M. Daneshtalab
Swarms of drones are increasingly been used in a variety of monitoring and surveillance, search and rescue, and photography and filming tasks. However, despite the growing popularity of swarm-based applications of drones, there is still a lack of approaches to generate efficient drone routes while minimizing the risks of drone collisions. In this paper, we present a novel approach that integrates learning, optimization, and prediction for generating efficient and safe routes for swarms of drones. The proposed approach comprises three main components: (1) a high-performance dynamic evolutionary algorithm for optimizing drone routes, (2) a reinforcement learning algorithm for incorporating the feedback and runtime data about the system state, and (3) a prediction approach to predict the movement of drones and moving obstacles in the flying zone. We also present a parallel implementation of the proposed approach and evaluate it against two benchmarks. The results demonstrate that the proposed approach allows to significantly reduce the route lengths and computation overhead while producing efficient and safe routes.
{"title":"Integrating Learning, Optimization, and Prediction for Efficient Navigation of Swarms of Drones","authors":"Amin Majd, A. Ashraf, E. Troubitsyna, M. Daneshtalab","doi":"10.1109/PDP2018.2018.00022","DOIUrl":"https://doi.org/10.1109/PDP2018.2018.00022","url":null,"abstract":"Swarms of drones are increasingly been used in a variety of monitoring and surveillance, search and rescue, and photography and filming tasks. However, despite the growing popularity of swarm-based applications of drones, there is still a lack of approaches to generate efficient drone routes while minimizing the risks of drone collisions. In this paper, we present a novel approach that integrates learning, optimization, and prediction for generating efficient and safe routes for swarms of drones. The proposed approach comprises three main components: (1) a high-performance dynamic evolutionary algorithm for optimizing drone routes, (2) a reinforcement learning algorithm for incorporating the feedback and runtime data about the system state, and (3) a prediction approach to predict the movement of drones and moving obstacles in the flying zone. We also present a parallel implementation of the proposed approach and evaluate it against two benchmarks. The results demonstrate that the proposed approach allows to significantly reduce the route lengths and computation overhead while producing efficient and safe routes.","PeriodicalId":333367,"journal":{"name":"2018 26th Euromicro International Conference on Parallel, Distributed and Network-based Processing (PDP)","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2018-03-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"122347929","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2018-03-21DOI: 10.1109/PDP2018.2018.00094
M. Ianni, E. Masciari, G. Mazzeo, C. Zaniolo
The need to support advanced analytics on Big Data is driving data scientist' interest toward massively parallel distributed systems and software platforms, such as Map- Reduce and Spark, that make possible their scalable utilization. However, when complex data mining algorithms are required, their fully scalable deployment on such platforms faces a number of technical challenges that grow with the complexity of the algorithms involved. Thus algorithms, that were originally designed for a sequential nature, must often be redesigned in order to effectively use the distributed computational resources. In this paper, we explore these problems, and then propose a solution which has proven to be very effective on the complex hierarchical clustering algorithm CLUBS+. We present a parallel version of CLUBS+ named CLUBS-P with an ad-hoc implementation based on message passing: CLUBS-MP.
{"title":"Clustering Goes Big: CLUBS-P, an Algorithm for Unsupervised Clustering Around Centroids Tailored For Big Data Applications","authors":"M. Ianni, E. Masciari, G. Mazzeo, C. Zaniolo","doi":"10.1109/PDP2018.2018.00094","DOIUrl":"https://doi.org/10.1109/PDP2018.2018.00094","url":null,"abstract":"The need to support advanced analytics on Big Data is driving data scientist' interest toward massively parallel distributed systems and software platforms, such as Map- Reduce and Spark, that make possible their scalable utilization. However, when complex data mining algorithms are required, their fully scalable deployment on such platforms faces a number of technical challenges that grow with the complexity of the algorithms involved. Thus algorithms, that were originally designed for a sequential nature, must often be redesigned in order to effectively use the distributed computational resources. In this paper, we explore these problems, and then propose a solution which has proven to be very effective on the complex hierarchical clustering algorithm CLUBS+. We present a parallel version of CLUBS+ named CLUBS-P with an ad-hoc implementation based on message passing: CLUBS-MP.","PeriodicalId":333367,"journal":{"name":"2018 26th Euromicro International Conference on Parallel, Distributed and Network-based Processing (PDP)","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2018-03-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"123006456","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2018-03-21DOI: 10.1109/PDP2018.2018.00109
P. Millar, Olufemi Adeyemi, G. Behrmann, P. Fuhrmann, V. Garonne, Dmitry Litvinsev, T. Mkrtchyan, A. Rossi, M. Sahakyan, Jürgen Starek
The dCache project provides open-source storage software deployed internationally to satisfy ever more demanding scientific storage requirements. Its multifaceted approach provides an integrated way of supporting different use-cases with the same storage, from high throughput data ingest, through wide access and easy integration with existing systems. In this paper, we describe some of the recent features that facilitate the use of storage to maximise the gain from stored data, including quality-of-service management, heterogeneous systems — both through integrated tertiary storage support and geographical locality — the parallel NFS (pNFS) extension, and innovative delegated authorisation schemes.
{"title":"Storage for Advanced Scientific Use-Cases and Beyond","authors":"P. Millar, Olufemi Adeyemi, G. Behrmann, P. Fuhrmann, V. Garonne, Dmitry Litvinsev, T. Mkrtchyan, A. Rossi, M. Sahakyan, Jürgen Starek","doi":"10.1109/PDP2018.2018.00109","DOIUrl":"https://doi.org/10.1109/PDP2018.2018.00109","url":null,"abstract":"The dCache project provides open-source storage software deployed internationally to satisfy ever more demanding scientific storage requirements. Its multifaceted approach provides an integrated way of supporting different use-cases with the same storage, from high throughput data ingest, through wide access and easy integration with existing systems. In this paper, we describe some of the recent features that facilitate the use of storage to maximise the gain from stored data, including quality-of-service management, heterogeneous systems — both through integrated tertiary storage support and geographical locality — the parallel NFS (pNFS) extension, and innovative delegated authorisation schemes.","PeriodicalId":333367,"journal":{"name":"2018 26th Euromicro International Conference on Parallel, Distributed and Network-based Processing (PDP)","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2018-03-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"121680676","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2018-03-16DOI: 10.1109/PDP2018.2018.00116
Md. Shahriar Karim, H. Othmer, David M. Umulis
Reaction-diffusion (RD) models are widely used to study the spatio-temporal evolution of pattern formation during development. Nonlinear RD models are often analytically intractable, and require numerical solution methods. Interrogation of RD models for a large physiological range of parameters covers many orders of magnitude, establishing situations where solutions are stiff and solvers fail to provide accurate results to the time-dependent problem. The spatial dependence of these parameters, and the nonlinearity of the underlying dynamics, impose additional challenges. We developed an efficient approach for simulating stiff RD models of pattern formation and we used supercomputer clusters to carry out a large screen of spatially varying parameters. The proposed approach generated data for screening of RD systems within a reasonable amount of time (a few days), which scales down further if additional cluster nodes are available. The approaches outlined herein are applicable to any systems biology problem requiring numerical approximation of RD equations with spatially non-uniform properties and stiff nonlinear reactions.
{"title":"Leveraging Compute Clusters for Large-Scale Parametric Screens of Reaction-Diffusion Systems","authors":"Md. Shahriar Karim, H. Othmer, David M. Umulis","doi":"10.1109/PDP2018.2018.00116","DOIUrl":"https://doi.org/10.1109/PDP2018.2018.00116","url":null,"abstract":"Reaction-diffusion (RD) models are widely used to study the spatio-temporal evolution of pattern formation during development. Nonlinear RD models are often analytically intractable, and require numerical solution methods. Interrogation of RD models for a large physiological range of parameters covers many orders of magnitude, establishing situations where solutions are stiff and solvers fail to provide accurate results to the time-dependent problem. The spatial dependence of these parameters, and the nonlinearity of the underlying dynamics, impose additional challenges. We developed an efficient approach for simulating stiff RD models of pattern formation and we used supercomputer clusters to carry out a large screen of spatially varying parameters. The proposed approach generated data for screening of RD systems within a reasonable amount of time (a few days), which scales down further if additional cluster nodes are available. The approaches outlined herein are applicable to any systems biology problem requiring numerical approximation of RD equations with spatially non-uniform properties and stiff nonlinear reactions.","PeriodicalId":333367,"journal":{"name":"2018 26th Euromicro International Conference on Parallel, Distributed and Network-based Processing (PDP)","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2018-03-16","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"116219934","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2018-03-01DOI: 10.1109/PDP2018.2018.00025
Aurelio Nicolas Mata, N. P. Castellanos-Abrego, G. Román-Alonso, M. Castro-García, G. Garza, J. R. Godínez-Fernández
The sinoatrial node (SAN) has the function of initiating a depolarizing wave that runs throughout the heart. This wave produces the muscular contraction necessary to blood pumping in animals. In recent years different works have been proposed to simulate the electric potential behaviour of a single sinoatrial cell (SANC) and groups of cells, hence a set of differential equations needs to be solved for each microsecond of simulation. An important drawback comes up when the synchronization of millions of SANCs is required involving a huge processing time. Since the simulation of the behavior of a set of cells is an open research topic, it is important to propose efficient tools to reduce response times; unfortunately, because the complexity of the existent models of SANC, very scarce work has been done to this end. This paper proposes three parallel algorithms to simulate the synchronization of a set of SANCs based on the model of Severi (2012). The proposed approaches are built using OpenMP, MPI, and CUDA, in order to compare the benefits given by different computing platforms. We found that all parallel versions perform better when defining a cell per processing unit; however the CUDA version gives the best results in scalability and performance.
{"title":"Parallel Simulation of Sinoatrial Node Cells Synchronization","authors":"Aurelio Nicolas Mata, N. P. Castellanos-Abrego, G. Román-Alonso, M. Castro-García, G. Garza, J. R. Godínez-Fernández","doi":"10.1109/PDP2018.2018.00025","DOIUrl":"https://doi.org/10.1109/PDP2018.2018.00025","url":null,"abstract":"The sinoatrial node (SAN) has the function of initiating a depolarizing wave that runs throughout the heart. This wave produces the muscular contraction necessary to blood pumping in animals. In recent years different works have been proposed to simulate the electric potential behaviour of a single sinoatrial cell (SANC) and groups of cells, hence a set of differential equations needs to be solved for each microsecond of simulation. An important drawback comes up when the synchronization of millions of SANCs is required involving a huge processing time. Since the simulation of the behavior of a set of cells is an open research topic, it is important to propose efficient tools to reduce response times; unfortunately, because the complexity of the existent models of SANC, very scarce work has been done to this end. This paper proposes three parallel algorithms to simulate the synchronization of a set of SANCs based on the model of Severi (2012). The proposed approaches are built using OpenMP, MPI, and CUDA, in order to compare the benefits given by different computing platforms. We found that all parallel versions perform better when defining a cell per processing unit; however the CUDA version gives the best results in scalability and performance.","PeriodicalId":333367,"journal":{"name":"2018 26th Euromicro International Conference on Parallel, Distributed and Network-based Processing (PDP)","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2018-03-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"128425985","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2018-03-01DOI: 10.1109/PDP2018.2018.00034
Ernesto Dufrechu, P. Ezzatti
Sparse triangular linear systems are ubiquitous in a wide range of science and engineering fields, and represent one of the most important building blocks of Sparse Numerical Lineal Algebra methods. For this reason, their parallel solution has been subject of exhaustive study, and efficient implementations of this kernel can be found for almost every hardware platform. However, the strong data dependencies that serialize a great deal of the execution and the load imbalance inherent to the triangular structure poses serious difficulties for its parallel performance, specially in the context of massively- parallel processors such as GPUs. To this day, the most widespread GPU implementation of this kernel is the one distributed in NVIDIA CUSPARSE library, which relies on a preprocessing stage to determine the parallel execution schedule. Although the solution phase is highly efficient, this strategy pays the cost of constant synchronizations with the CPU. In this work, we present a synchronization-free GPU al- gorithm to solve sparse triangular linear systems for the CSR format. The experimental evaluation shows performance improvements over CUSPARSE and a recently proposed synchronization-free method for the CSC matrix format.
{"title":"Solving Sparse Triangular Linear Systems in Modern GPUs: A Synchronization-Free Algorithm","authors":"Ernesto Dufrechu, P. Ezzatti","doi":"10.1109/PDP2018.2018.00034","DOIUrl":"https://doi.org/10.1109/PDP2018.2018.00034","url":null,"abstract":"Sparse triangular linear systems are ubiquitous in a wide range of science and engineering fields, and represent one of the most important building blocks of Sparse Numerical Lineal Algebra methods. For this reason, their parallel solution has been subject of exhaustive study, and efficient implementations of this kernel can be found for almost every hardware platform. However, the strong data dependencies that serialize a great deal of the execution and the load imbalance inherent to the triangular structure poses serious difficulties for its parallel performance, specially in the context of massively- parallel processors such as GPUs. To this day, the most widespread GPU implementation of this kernel is the one distributed in NVIDIA CUSPARSE library, which relies on a preprocessing stage to determine the parallel execution schedule. Although the solution phase is highly efficient, this strategy pays the cost of constant synchronizations with the CPU. In this work, we present a synchronization-free GPU al- gorithm to solve sparse triangular linear systems for the CSR format. The experimental evaluation shows performance improvements over CUSPARSE and a recently proposed synchronization-free method for the CSC matrix format.","PeriodicalId":333367,"journal":{"name":"2018 26th Euromicro International Conference on Parallel, Distributed and Network-based Processing (PDP)","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2018-03-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"132147320","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2018-03-01DOI: 10.1109/PDP2018.2018.00106
L. Morganti, D. Cesini, Elena Corni, Luca Lama, Carmelo Pellegrino, I. Merelli, D. D'Agostino
Low-power Systems on Chip (SoCs) derived from the embedded and mobile market can be profitably used to execute scientific workloads traditionally designed for power-hungry clusters, saving energy, gaining portability and reducing infrastructural costs and sizes. We investigate the possibility of using SoCs as storage bricks of a BeeGFS filesystem in the perspective of energy-efficient storage solutions supporting scientific computing. Then, we consider a use case from metagenomics analysis and show how the large amount of genome sequencing information streamed by portable sequencing devices could be managed by low-power SoCs making use of an underlying BeeGFS filesystem.
{"title":"Low-Power Storage Bricks and Bioinformatics on Systems-On-Chip","authors":"L. Morganti, D. Cesini, Elena Corni, Luca Lama, Carmelo Pellegrino, I. Merelli, D. D'Agostino","doi":"10.1109/PDP2018.2018.00106","DOIUrl":"https://doi.org/10.1109/PDP2018.2018.00106","url":null,"abstract":"Low-power Systems on Chip (SoCs) derived from the embedded and mobile market can be profitably used to execute scientific workloads traditionally designed for power-hungry clusters, saving energy, gaining portability and reducing infrastructural costs and sizes. We investigate the possibility of using SoCs as storage bricks of a BeeGFS filesystem in the perspective of energy-efficient storage solutions supporting scientific computing. Then, we consider a use case from metagenomics analysis and show how the large amount of genome sequencing information streamed by portable sequencing devices could be managed by low-power SoCs making use of an underlying BeeGFS filesystem.","PeriodicalId":333367,"journal":{"name":"2018 26th Euromicro International Conference on Parallel, Distributed and Network-based Processing (PDP)","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2018-03-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"126216077","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2018-03-01DOI: 10.1109/PDP2018.2018.00087
A. Marosi, Attila Farkas, R. Lovas
Internet of Things (IoT) is playing increasingly more fundamental role in wide range of sectors, including industry, agriculture, health care, and other services. In many cases, cloud computing serves as an elastic and efficient paradigm for implementing IoT back-ends. With the emerging lightweight software container technologies, the feasible approaches and design options for such IoT back-ends have been significantly enriched. In our paper we present the evolution of an IoT back-end, which is responsible for collecting (among others) meteorological, image and soil data from cultivated fields in order to enable precision farming. The different versions, namely the cloud VM-based and the Docker containerized variants, provide highly scalable and vendor independent (cloud provider agnostic) solutions, therefore they can form a robust and adaptive framework for further pilot applications areas, e.g. Connected Cars and Industry 4.0, as the presented benchmarks illustrate the throughput and other parameters of the current implementation in the paper.
{"title":"An Adaptive Cloud-Based IoT Back-end Architecture and Its Applications","authors":"A. Marosi, Attila Farkas, R. Lovas","doi":"10.1109/PDP2018.2018.00087","DOIUrl":"https://doi.org/10.1109/PDP2018.2018.00087","url":null,"abstract":"Internet of Things (IoT) is playing increasingly more fundamental role in wide range of sectors, including industry, agriculture, health care, and other services. In many cases, cloud computing serves as an elastic and efficient paradigm for implementing IoT back-ends. With the emerging lightweight software container technologies, the feasible approaches and design options for such IoT back-ends have been significantly enriched. In our paper we present the evolution of an IoT back-end, which is responsible for collecting (among others) meteorological, image and soil data from cultivated fields in order to enable precision farming. The different versions, namely the cloud VM-based and the Docker containerized variants, provide highly scalable and vendor independent (cloud provider agnostic) solutions, therefore they can form a robust and adaptive framework for further pilot applications areas, e.g. Connected Cars and Industry 4.0, as the presented benchmarks illustrate the throughput and other parameters of the current implementation in the paper.","PeriodicalId":333367,"journal":{"name":"2018 26th Euromicro International Conference on Parallel, Distributed and Network-based Processing (PDP)","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2018-03-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"121777029","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2018-03-01DOI: 10.1109/PDP2018.2018.00108
Christos Filippidis, Y. Cotronis
Experiments using the Large Hadron Collider (LHC) currently generate tens of petabytes of reduced data per year, observational and simulation data in the climate domain is expected to reach eXabytes by 2021, and light source experiments are expected to generate hundreds of terabytes per day. At such extreme scale, the substantial amount of concurrency can cause critical contention issue of the I/O system. This study introduces pWebDAV as a heterogeneous, multi-tier storage system. pWebDAV proposes a dynamically coordinated I/O architecture offering overall data flow solutions (remote-local access). The fundamental idea is to implement, for each data transfer, I/O policies on the fly. pWebDAV controls all I/O nodes, participating in the data transfer, directly regardless of the tier. pWebDAV approach can fully utilize the provided I/O & network resources and is able to minimize disk and network contention. The focus in this study is the Metadata node scalability performance.
{"title":"pWebDAV: A Multi-Tier Storage System","authors":"Christos Filippidis, Y. Cotronis","doi":"10.1109/PDP2018.2018.00108","DOIUrl":"https://doi.org/10.1109/PDP2018.2018.00108","url":null,"abstract":"Experiments using the Large Hadron Collider (LHC) currently generate tens of petabytes of reduced data per year, observational and simulation data in the climate domain is expected to reach eXabytes by 2021, and light source experiments are expected to generate hundreds of terabytes per day. At such extreme scale, the substantial amount of concurrency can cause critical contention issue of the I/O system. This study introduces pWebDAV as a heterogeneous, multi-tier storage system. pWebDAV proposes a dynamically coordinated I/O architecture offering overall data flow solutions (remote-local access). The fundamental idea is to implement, for each data transfer, I/O policies on the fly. pWebDAV controls all I/O nodes, participating in the data transfer, directly regardless of the tier. pWebDAV approach can fully utilize the provided I/O & network resources and is able to minimize disk and network contention. The focus in this study is the Metadata node scalability performance.","PeriodicalId":333367,"journal":{"name":"2018 26th Euromicro International Conference on Parallel, Distributed and Network-based Processing (PDP)","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2018-03-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"130573824","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：