Performance Evaluation for Real-Time Messaging System in Big Data Pipeline Architecture

Abstract

Nowadays, Real time messaging system is the essential thing in enabling time-critical decision making in many applications where it is important to deal with real-time requirements and reliability requirements simultaneously. For dependability reasons, we intend to maximize the reliability requirement of real time messaging system. To develop real time messaging system, we create real time big data pipeline by using Apache Kafka and Apache Storm. This paper focuses on analyzing the performance of producer and consumer in Apache Kafka processing. The performance of Kafka processing modify to be more reliable on the pipeline architecture. Then, the experiment will be conducted the processing time in the performance of the producer and consumer on various partitions. The performance evaluation of Kafka can impact on messaging system in real time big data pipeline architecture.