Real-time data analysis using Spark and Hadoop

Abstract

Big Data is at use every single day, from the adoption of Internet through social networks, mobile devices, connected objects, videos, blogs and others. Big Data real-time processing have received a growing attention especially with the expansion of data in volume and complexity. Big data is created every day, from the use of the Internet through social networks, mobile devices, connected objects, videos, blogs and others. In order to ensure a reliable and a fast real-time information processing, powerful tools are essential for the analysis and processing of Big Data. Standards MapReduce frameworks such as Hadoop MapReduce face some limitations for processing real-time data of various formats. In this paper, we highlight the implementation of the de-facto standard Hadoop MapReduce and also the implementation of the framework Apache Spark. Thereafter, we conduct experimental simulations to analyze a real-time data stream using Spark and Hadoop. To further enforce our contribution, we introduce a comparison of the two implementations in terms of architecture and performance with a discussion to feature the results of simulations. The paper discusses also the drawbacks of using Hadoop for real-time processing.