Pub Date : 2015-07-20DOI: 10.1109/HPCSim.2015.7237082
Md. Mohsin Ali, P. Strazdins, B. Harding, M. Hegland, J. Larson
Applications performing ultra-large scale simulations via solving PDEs require very large computational systems for their timely solution. Studies have shown the rate of failure grows with the system size and these trends are likely to worsen in future machines as less reliable components are used to reduce the energy cost. Thus, as systems, and the problems solved on them, continue to grow, the ability to survive failures is becoming a critical aspect of algorithm development. The sparse grid combination technique (SGCT) is a cost-effective method for solving time-evolving PDEs, especially for higher-dimensional problems. It can also be easily modified to provide algorithm-based fault tolerance for these problems. In this paper, we show how the SGCT can produce a fault-tolerant version of the GENE gyrokinetic plasma application, which evolves a 5D complex density field over time. We use an alternate component grid combination formula to recover data from lost processes. User Level Failure Mitigation (ULFM) MPI is used to recover the processes, and our implementation is robust over multiple failures and recovery for both process and node failures. An acceptable degree of modification of the application is required. Results using the SGCT on two of the fields' dimensions show competitive execution times with acceptable error (within 0.1%), compared to the same simulation with a single full resolution grid. The benefits improve when the SGCT is used over three dimensions. Our experiments show that the GENE application can successfully recover from multiple process failures, and applying the SGCT the corresponding number of times minimizes the error for the lost sub-grids. Application recovery overhead via ULFM MPI increases from ~1.5s at 64 cores to ~5s at 2048 cores for a one-off failure. This compares favourably to using GENE's in-built checkpointing with job restart in conjunction with the classical SGCT on failure, which have overheads four times as large for a single failure, excluding the backtrack overhead. An analysis for a long-running application taking into account checkpoint backtrack times indicates a reduction in overhead of over an order of magnitude.
{"title":"A fault-tolerant gyrokinetic plasma application using the sparse grid combination technique","authors":"Md. Mohsin Ali, P. Strazdins, B. Harding, M. Hegland, J. Larson","doi":"10.1109/HPCSim.2015.7237082","DOIUrl":"https://doi.org/10.1109/HPCSim.2015.7237082","url":null,"abstract":"Applications performing ultra-large scale simulations via solving PDEs require very large computational systems for their timely solution. Studies have shown the rate of failure grows with the system size and these trends are likely to worsen in future machines as less reliable components are used to reduce the energy cost. Thus, as systems, and the problems solved on them, continue to grow, the ability to survive failures is becoming a critical aspect of algorithm development. The sparse grid combination technique (SGCT) is a cost-effective method for solving time-evolving PDEs, especially for higher-dimensional problems. It can also be easily modified to provide algorithm-based fault tolerance for these problems. In this paper, we show how the SGCT can produce a fault-tolerant version of the GENE gyrokinetic plasma application, which evolves a 5D complex density field over time. We use an alternate component grid combination formula to recover data from lost processes. User Level Failure Mitigation (ULFM) MPI is used to recover the processes, and our implementation is robust over multiple failures and recovery for both process and node failures. An acceptable degree of modification of the application is required. Results using the SGCT on two of the fields' dimensions show competitive execution times with acceptable error (within 0.1%), compared to the same simulation with a single full resolution grid. The benefits improve when the SGCT is used over three dimensions. Our experiments show that the GENE application can successfully recover from multiple process failures, and applying the SGCT the corresponding number of times minimizes the error for the lost sub-grids. Application recovery overhead via ULFM MPI increases from ~1.5s at 64 cores to ~5s at 2048 cores for a one-off failure. This compares favourably to using GENE's in-built checkpointing with job restart in conjunction with the classical SGCT on failure, which have overheads four times as large for a single failure, excluding the backtrack overhead. An analysis for a long-running application taking into account checkpoint backtrack times indicates a reduction in overhead of over an order of magnitude.","PeriodicalId":134009,"journal":{"name":"2015 International Conference on High Performance Computing & Simulation (HPCS)","volume":"5 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2015-07-20","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"131359520","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2015-07-20DOI: 10.1109/HPCSim.2015.7237076
J. Falcou
Numerical simulations running on computers is the most fundamental tool that most sciences - from physics to social science - use as a substitute to experiments when said experiments cannot realistically be run with a satisfactory duration, budget or ethical framework. This also means that the accuracy and the speed at which such computer simulations can be done is a crucial factor for the global scientific advancement. If accuracy of the simulation is tied to the field knowledge of scientists, the speed of a simulation is tied to the way one may take advantage of a computer hardware.
{"title":"Designing HPC libraries in the modern C++ world","authors":"J. Falcou","doi":"10.1109/HPCSim.2015.7237076","DOIUrl":"https://doi.org/10.1109/HPCSim.2015.7237076","url":null,"abstract":"Numerical simulations running on computers is the most fundamental tool that most sciences - from physics to social science - use as a substitute to experiments when said experiments cannot realistically be run with a satisfactory duration, budget or ethical framework. This also means that the accuracy and the speed at which such computer simulations can be done is a crucial factor for the global scientific advancement. If accuracy of the simulation is tied to the field knowledge of scientists, the speed of a simulation is tied to the way one may take advantage of a computer hardware.","PeriodicalId":134009,"journal":{"name":"2015 International Conference on High Performance Computing & Simulation (HPCS)","volume":"52 ","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2015-07-20","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"120886852","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2015-07-20DOI: 10.1109/HPCSim.2015.7237091
L. Indrusiak
This paper summarises the latest developments on the resource allocation of real-time and mixed-criticality applications onto multiprocessor platforms based on Networks-on-Chip. The paper focuses on priority-preemptive Networks-on-Chip, and on the use of analytical models as fitness functions to guide search-based heuristics towards fully schedulable allocations, therefore suitable for mixed criticality applications with hard real-time constraints.
{"title":"Real-time mixed-criticality Network-on-Chip resource allocation","authors":"L. Indrusiak","doi":"10.1109/HPCSim.2015.7237091","DOIUrl":"https://doi.org/10.1109/HPCSim.2015.7237091","url":null,"abstract":"This paper summarises the latest developments on the resource allocation of real-time and mixed-criticality applications onto multiprocessor platforms based on Networks-on-Chip. The paper focuses on priority-preemptive Networks-on-Chip, and on the use of analytical models as fitness functions to guide search-based heuristics towards fully schedulable allocations, therefore suitable for mixed criticality applications with hard real-time constraints.","PeriodicalId":134009,"journal":{"name":"2015 International Conference on High Performance Computing & Simulation (HPCS)","volume":"2 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2015-07-20","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"127995014","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2015-07-20DOI: 10.1109/HPCSim.2015.7237070
Ashutosh Kumar Singh, P. Dziurzański, L. Indrusiak
Many-core systems are envisioned to fulfill the increased performance demands in several computing domains such as embedded and high performance computing (HPC). The HPC systems are often overloaded to execute a number of dynamically arriving jobs. In overload situations, market-inspired resource allocation heuristics have been found to provide better results in terms of overall profit (value) earned by completing the execution of a number of jobs when compared to various other heuristics. However, the conventional market-inspired heuristics lack the concept of holding low value executing jobs to free the occupied resources to be used by high value arrived jobs in order to maximize the overall profit. In this paper, we propose a market-inspired heuristic that accomplish the aforementioned concept and utilizes design-time profiling results of jobs to facilitate efficient allocation. Additionally, the remaining executions of the held jobs are performed on freed resources at later stages to make some profit out of them. The holding process identifies the appropriate jobs to be put on hold to free the resources and ensures that the loss incurred due to holding is lower than the profit achieved by high value arrived jobs by using the free resources. Experiments show that the proposed approach achieves 8% higher savings when compared to existing approaches, which can be a significant amount when dealing in the order of millions of dollars.
{"title":"Market-inspired dynamic resource allocation in many-core high performance computing systems","authors":"Ashutosh Kumar Singh, P. Dziurzański, L. Indrusiak","doi":"10.1109/HPCSim.2015.7237070","DOIUrl":"https://doi.org/10.1109/HPCSim.2015.7237070","url":null,"abstract":"Many-core systems are envisioned to fulfill the increased performance demands in several computing domains such as embedded and high performance computing (HPC). The HPC systems are often overloaded to execute a number of dynamically arriving jobs. In overload situations, market-inspired resource allocation heuristics have been found to provide better results in terms of overall profit (value) earned by completing the execution of a number of jobs when compared to various other heuristics. However, the conventional market-inspired heuristics lack the concept of holding low value executing jobs to free the occupied resources to be used by high value arrived jobs in order to maximize the overall profit. In this paper, we propose a market-inspired heuristic that accomplish the aforementioned concept and utilizes design-time profiling results of jobs to facilitate efficient allocation. Additionally, the remaining executions of the held jobs are performed on freed resources at later stages to make some profit out of them. The holding process identifies the appropriate jobs to be put on hold to free the resources and ensures that the loss incurred due to holding is lower than the profit achieved by high value arrived jobs by using the free resources. Experiments show that the proposed approach achieves 8% higher savings when compared to existing approaches, which can be a significant amount when dealing in the order of millions of dollars.","PeriodicalId":134009,"journal":{"name":"2015 International Conference on High Performance Computing & Simulation (HPCS)","volume":"14 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2015-07-20","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"127459653","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2015-07-20DOI: 10.1109/HPCSim.2015.7237108
Mikolaj Baranowski, M. Bubak, A. Belloum
Applying suitable application models is essential to achieve efficient execution of the applications, effective development process and assure application portability and reusability. We observe that execution environments and the cloud environment in particular, lack of tools that would make it more available from a programmer perspective. Complexity and variety of libraries and authentication methods do not correspond with simplistic functionality of those services. We approach this issue at several fields - workflow systems, domain specific languages and cloud computing. Our work focus on two points, optimizing existing application models by supplementing them with additional tools and providing new features, and develop new models that are based on our research on applications in the cloud environment.
{"title":"Data and process abstractions for cloud computing","authors":"Mikolaj Baranowski, M. Bubak, A. Belloum","doi":"10.1109/HPCSim.2015.7237108","DOIUrl":"https://doi.org/10.1109/HPCSim.2015.7237108","url":null,"abstract":"Applying suitable application models is essential to achieve efficient execution of the applications, effective development process and assure application portability and reusability. We observe that execution environments and the cloud environment in particular, lack of tools that would make it more available from a programmer perspective. Complexity and variety of libraries and authentication methods do not correspond with simplistic functionality of those services. We approach this issue at several fields - workflow systems, domain specific languages and cloud computing. Our work focus on two points, optimizing existing application models by supplementing them with additional tools and providing new features, and develop new models that are based on our research on applications in the cloud environment.","PeriodicalId":134009,"journal":{"name":"2015 International Conference on High Performance Computing & Simulation (HPCS)","volume":"118 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2015-07-20","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"130968375","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2015-07-20DOI: 10.1109/HPCSim.2015.7237097
Rossella Arcucci, L. D’Amore, L. Carracciuolo
We present an innovative approach for solving Four Dimensional Variational Data Assimilation (4D-VAR DA) problems. The approach we consider starts from a decomposition of the physical domain; it uses a partitioning of the solution and a modified regularization functional describing the 4D-VAR DA problem on the decomposition. We provide a mathematical formulation of the model and we perform a feasibility analysis in terms of computational cost and of algorithmic scalability. We use the scale-up factor which measure the performance gain in terms of time complexity reduction. We verify the reliability of the approach on a consistent test case (the Shallow Water Equations).
{"title":"On the problem-decomposition of scalable 4D-Var Data Assimilation models","authors":"Rossella Arcucci, L. D’Amore, L. Carracciuolo","doi":"10.1109/HPCSim.2015.7237097","DOIUrl":"https://doi.org/10.1109/HPCSim.2015.7237097","url":null,"abstract":"We present an innovative approach for solving Four Dimensional Variational Data Assimilation (4D-VAR DA) problems. The approach we consider starts from a decomposition of the physical domain; it uses a partitioning of the solution and a modified regularization functional describing the 4D-VAR DA problem on the decomposition. We provide a mathematical formulation of the model and we perform a feasibility analysis in terms of computational cost and of algorithmic scalability. We use the scale-up factor which measure the performance gain in terms of time complexity reduction. We verify the reliability of the approach on a consistent test case (the Shallow Water Equations).","PeriodicalId":134009,"journal":{"name":"2015 International Conference on High Performance Computing & Simulation (HPCS)","volume":"62 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2015-07-20","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"131148436","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2015-07-20DOI: 10.1109/HPCSim.2015.7237020
J. Filipovič, Jan Plhak, D. Střelák
In this paper, we introduce the GPU acceleration of dRMSD algorithm, used to compare different structures of a molecule. Comparing to multithreaded CPU implementation, we have reached 13.4× speedup in clustering and 62.7× speedup in I:I dRMSD computation using mid-end GPU. The dRMSD computation exposes strong memory locality and thus is compute-bound. Along with conservative implementation using shared memory, we have decided to implement variants of the algorithm using GPU caches to maintain memory locality. Our implementation using cache reaches 96.5% and 91.6% of shared memory performance on Fermi and Maxwell, respectively. We have identified several performance pitfalls related to cache blocking in compute-bound codes and suggested optimization techniques to improve the performance.
{"title":"Acceleration of dRMSD calculation and efficient usage of GPU caches","authors":"J. Filipovič, Jan Plhak, D. Střelák","doi":"10.1109/HPCSim.2015.7237020","DOIUrl":"https://doi.org/10.1109/HPCSim.2015.7237020","url":null,"abstract":"In this paper, we introduce the GPU acceleration of dRMSD algorithm, used to compare different structures of a molecule. Comparing to multithreaded CPU implementation, we have reached 13.4× speedup in clustering and 62.7× speedup in I:I dRMSD computation using mid-end GPU. The dRMSD computation exposes strong memory locality and thus is compute-bound. Along with conservative implementation using shared memory, we have decided to implement variants of the algorithm using GPU caches to maintain memory locality. Our implementation using cache reaches 96.5% and 91.6% of shared memory performance on Fermi and Maxwell, respectively. We have identified several performance pitfalls related to cache blocking in compute-bound codes and suggested optimization techniques to improve the performance.","PeriodicalId":134009,"journal":{"name":"2015 International Conference on High Performance Computing & Simulation (HPCS)","volume":"103 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2015-07-20","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"132728249","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2015-07-20DOI: 10.1109/HPCSim.2015.7237051
Chen Chen, Qingqi Pei
In this paper, we investigate the self-similarity characteristic of MANET(Mobile Ad-hoc NETworks) traffics through simulations and then construct a fuzzy logic controlled mobility model according to the traffic feature to optimize the network performance. First, based on the generated traffics using OPNET, the self-similarity of MANET traffics has been verified with a qualitative analysis. Then, by exploring the relation between the self-similarity indicator, i.e., Hurst and some network performance metrics, such as Packets Delivery Ratio(PDR), Average Transmission Delay(ATD) and nodal Average Moving Speed(AMS), a fuzzy logic controller is designed to make the mobility model adaptively work in order to output satisfied performance. By online estimating the self-similarity of incoming traffics using R/S analysis, the Packet Size(PS) and AMS of each vehicle could be intelligently adjusted to maximize the PDR and minimize the experienced ATD. Numerical results indicate that our proposed mobility model, compared to the classic mobility model RWP(Random WayPoint), has a better performance in terms of PDR and ATD.
{"title":"A fuzzy logic controlled mobility model based on simulated traffics' characteristics in MANET","authors":"Chen Chen, Qingqi Pei","doi":"10.1109/HPCSim.2015.7237051","DOIUrl":"https://doi.org/10.1109/HPCSim.2015.7237051","url":null,"abstract":"In this paper, we investigate the self-similarity characteristic of MANET(Mobile Ad-hoc NETworks) traffics through simulations and then construct a fuzzy logic controlled mobility model according to the traffic feature to optimize the network performance. First, based on the generated traffics using OPNET, the self-similarity of MANET traffics has been verified with a qualitative analysis. Then, by exploring the relation between the self-similarity indicator, i.e., Hurst and some network performance metrics, such as Packets Delivery Ratio(PDR), Average Transmission Delay(ATD) and nodal Average Moving Speed(AMS), a fuzzy logic controller is designed to make the mobility model adaptively work in order to output satisfied performance. By online estimating the self-similarity of incoming traffics using R/S analysis, the Packet Size(PS) and AMS of each vehicle could be intelligently adjusted to maximize the PDR and minimize the experienced ATD. Numerical results indicate that our proposed mobility model, compared to the classic mobility model RWP(Random WayPoint), has a better performance in terms of PDR and ATD.","PeriodicalId":134009,"journal":{"name":"2015 International Conference on High Performance Computing & Simulation (HPCS)","volume":"20 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2015-07-20","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"134541778","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2015-07-20DOI: 10.1109/HPCSim.2015.7237019
J. Bardino, Martin Rehr, B. Vinter
With the first version of the Cph CT Toolbox released and introduced, we turn to intensively evaluating the performance of the FDK and Katsevich reconstruction implementations in the second major release. The evaluation focuses on comparisons between different hardware platforms from the two major GPU compute vendors, AMD and NVIDIA, using our updated CUDA and new OpenCL implementations. Such a performance comparison is in itself interesting in a narrow CT scanning and reconstruction perspective, but it also sheds some light on the performance of those AMD and NVIDIA platforms and GPU technologies: something of general interest to anyone building or considering GPU solutions for their scientific calculations. Results from the best system reveals the chosen streaming strategy to scale linearly up to problem sizes one order of magnitude larger than the available GPU memory, and with only a minor scaling decrease when increasing the problem size further to the next order of magnitude.
{"title":"Cph CT Toolbox: A performance evaluation","authors":"J. Bardino, Martin Rehr, B. Vinter","doi":"10.1109/HPCSim.2015.7237019","DOIUrl":"https://doi.org/10.1109/HPCSim.2015.7237019","url":null,"abstract":"With the first version of the Cph CT Toolbox released and introduced, we turn to intensively evaluating the performance of the FDK and Katsevich reconstruction implementations in the second major release. The evaluation focuses on comparisons between different hardware platforms from the two major GPU compute vendors, AMD and NVIDIA, using our updated CUDA and new OpenCL implementations. Such a performance comparison is in itself interesting in a narrow CT scanning and reconstruction perspective, but it also sheds some light on the performance of those AMD and NVIDIA platforms and GPU technologies: something of general interest to anyone building or considering GPU solutions for their scientific calculations. Results from the best system reveals the chosen streaming strategy to scale linearly up to problem sizes one order of magnitude larger than the available GPU memory, and with only a minor scaling decrease when increasing the problem size further to the next order of magnitude.","PeriodicalId":134009,"journal":{"name":"2015 International Conference on High Performance Computing & Simulation (HPCS)","volume":"605 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2015-07-20","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"116381849","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2015-07-20DOI: 10.1109/HPCSim.2015.7237086
Irving Cordova, Teng-Sheng Moh
DBSCAN is a well-known density-based data clustering algorithm that is widely used due to its ability to find arbitrarily shaped clusters in noisy data. However, DBSCAN is hard to scale which limits its utility when working with large data sets. Resilient Distributed Datasets (RDDs), on the other hand, are a fast data-processing abstraction created explicitly for in-memory computation of large data sets. This paper presents a new algorithm based on DBSCAN using the Resilient Distributed Datasets approach: RDD-DBSCAN. RDD-DBSCAN overcomes the scalability limitations of the traditional DBSCAN algorithm by operating in a fully distributed fashion. The paper also evaluates an implementation of RDD-DBSCAN using Apache Spark, the official RDD implementation.
{"title":"DBSCAN on Resilient Distributed Datasets","authors":"Irving Cordova, Teng-Sheng Moh","doi":"10.1109/HPCSim.2015.7237086","DOIUrl":"https://doi.org/10.1109/HPCSim.2015.7237086","url":null,"abstract":"DBSCAN is a well-known density-based data clustering algorithm that is widely used due to its ability to find arbitrarily shaped clusters in noisy data. However, DBSCAN is hard to scale which limits its utility when working with large data sets. Resilient Distributed Datasets (RDDs), on the other hand, are a fast data-processing abstraction created explicitly for in-memory computation of large data sets. This paper presents a new algorithm based on DBSCAN using the Resilient Distributed Datasets approach: RDD-DBSCAN. RDD-DBSCAN overcomes the scalability limitations of the traditional DBSCAN algorithm by operating in a fully distributed fashion. The paper also evaluates an implementation of RDD-DBSCAN using Apache Spark, the official RDD implementation.","PeriodicalId":134009,"journal":{"name":"2015 International Conference on High Performance Computing & Simulation (HPCS)","volume":"40 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2015-07-20","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"117235355","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：