Exploring and Evaluating the Scalability and Eficinecy of Apache Spark Using Educational Datasets

Abstract

The combination of data mining and machine learning technology with web-based education system is becoming an imperative research area to enhance the quality of education beyond the traditional concept. With the worldwide fast growth of the Information Communication Technology (ICT), data come with significant large volume, high velocity and extensive variety. In this paper, four popular data mining methods are applied on Apache Spark using large volume of datasets from Online Cognitive Learning Systems to explore the scalability and efficiency of Spark. Various volumes of datasets are tested on Spark MLlib with different running configurations and parameter tunings. The output of the paper convincingly presents useful strategies of computing resource allocation and tuning to make full advantage of the in-memory system of Apache Spark with the tasks of data mining and machine learning on educational datasets.