Spark-based Distributed Intelligent Network Intrusion Detection System for Unified Dataset

Abstract

The proliferation of cloud computing is directly responsible for the current transformation phase that the information technology sector is going through. The concept of cloud computing is still in its infancy, yet it is altering the information technology industry. Due to the distributed and open nature of cloud services, they are vulnerable to various threats, including malicious activities and intrusions. Cloud services are also prone to be hacked. Conventional network intrusion detection systems (NIDS) are ineffective against today’s high-volume network traffic because they are trained using a single dataset. The infrastructure and application pose limitations, making processing enormous network traffic in real-time challenging. To protect the cloud from the numerous cloud-based dangers that exist, it is essential to embody Network intrusion detection systems (NIDS) which are equipped with intelligence. This research presents a solution to a modern problem: the development of a distributed and sophisticated NIDS framework using cloud-based solutions. An intelligent NIDS for cloud platforms is proposed in this article, along with an orchestration of a Docker-based Spark cluster over Kubernetes, which is hosted on AWS EC2 instances. The ANN-based NIDS that has been proposed attains an accuracy of 96.3% and encourages Precision scores of 97.2%, Recall scores of 97.5%, and F1-scores of 97.3%.