Comparative Analysis of Hadoop and Spark Performance for Real-time Big Data Smart Platforms Utilizing IoT Technology in Electrical Facilities

Abstract

As the adoption of IoT technology in power systems accelerates and the need for improved methods to handle large volumes of data emerges, real-time big data smart platforms must address the growing data processing demands in IoT integrated power systems. Therefore, in this study, we assess the performance of Hadoop and Spark for iterative computing and real-time data processing applications. Our evaluation is based on metrics such as execution time, resource utilization, and scalability, particularly with increasing data volume. The comparison aims to provide guidance to researchers, practitioners and entrepreneurs on platform selection depending on their specific requirements. The study identified the strengths and weaknesses of both platforms and provided valuable insights into optimizing the performance of big data applications. Text documents and charts for Word Count and PageRank tasks were used for comparison, and performance testing was performed on datasets of different sizes. The results showed that Spark outperforms Hadoop in most applications, especially in iterative computation and real-time data processing, due to its use of in-memory computation. However, Hadoop is best suited for batch processing operations that require multiple steps. It can perform these operations in parallel across multiple cluster nodes, enabling fast processing of large amounts of data. This comprehensive performance comparison of Hadoop and Spark in iterative computing and real-time data processing applications provides valuable information for researchers, practitioners, and enterprises on the trade-offs and benefits of using these big data platforms.