Performance Evaluation of Data Mining Frameworks in Hadoop Cluster Using Virtual Campus Log Files

Abstract

With the fast development in Cloud computing technologies, most computing platforms and stand alone applications are being deployed in Cloud platforms and offered as a service (SaaS). Likewise, Data Mining Frameworks (DMFs) such as Weka and R, are being ported to Cloud platforms, while other frameworks properly designed for Cloud platforms are emerging such as Mahout. For existing DMFs, which were designed before Cloud computing appeared, the main issue is if porting them to Cloud platforms would bring any benefits. One the one hand, by porting them to Cloud, it is possible to offer them as Cloud service, which would alleviate the final user from the burden of installing and configuring DMFs at local computer or local networking infrastructure. On the other hand, porting DMFs to Cloud should allow to tackle mining of very large data sets, i.e. Big Data. In this work we evaluate some DMFs, including Weka and Mahout, under a Hadoop cluster and show that while there are improvements in time efficiency to a certain scale, some mining functions, which are part of DMFs, were not able to finalize for data sets of more than 20Gb, namely, mining log files of a virtual campus. The study revealed that indeed porting DMFs to Cloud might not necessarily help tackling Big Data, as such DMFs were conceived without Big Data requirements.