Design of intelligent k-means based on spark for big data clustering

2016 International Workshop on Big Data and Information Security (IWBIS) Pub Date : 2016-10-01 DOI:10.1109/IWBIS.2016.7872895

Ilham Kusuma, M. A. Ma'sum, Novian Habibie, W. Jatmiko, H. Suhartanto

引用次数: 21

Abstract

The growth of data has bring us to the big data generation where the amount of data cannot be computed using conventional environment. There are a lot of computational environment that had been developed to compute big data, one of them is Hadoop that has Distributed File System and MapReduce framework. Spark is newly framework that can be combined with Hadoop and run on top of it. In this paper, we design intelligent k-means based on Spark for big data clustering. Our design is using batch of data instead using original Resilient Distributed Dataset (RDD). We compare our design with the implementation that using original RDD of data. Result of experiment shows that implementation using batch of data is faster than the implementation using original RDD.

查看原文

微信好友朋友圈 QQ好友复制链接

本刊更多论文

基于spark的大数据聚类智能k-means设计

数据的增长将我们带到了大数据的产生，而传统的环境无法计算数据量。目前已经开发了很多计算环境来计算大数据，其中之一就是拥有分布式文件系统和MapReduce框架的Hadoop。Spark是一个新的框架，可以与Hadoop结合并在其上运行。本文设计了基于Spark的大数据聚类智能k-means。我们的设计是使用批量数据，而不是使用原始的弹性分布式数据集(RDD)。我们将我们的设计与使用原始数据RDD的实现进行了比较。实验结果表明，使用批量数据的实现比使用原始RDD的实现速度更快。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文去求助

来源期刊

2016 International Workshop on Big Data and Information Security (IWBIS)

自引率

0.00%

发文量

期刊最新文献

Advancing public health genomics Overview of research center for information technology innovation in Taiwan Academia Sinica A survey of whole genome alignment tools and frameworks based on Hadoop's MapReduce Design and implementation of merchant acquirer data warehouse at PT. XYZ Spatial data mining for predicting of unobserved zinc pollutant using ordinary point Kriging