Strengthening cybersecurity: TestCloudIDS dataset and SparkShield algorithm for robust threat detection

A significant challenge in cybersecurity is the lack of a large-scale network dataset that accurately records modern traffic patterns, a wide variety of modest incursions, and comprehensive network traffic data. Existing benchmark datasets such as KDDCup99, NSL-KDD, GureKDD, and UNSWNB-15 must be updated to reflect modern cyber attack signatures. To address this issue, a new labeled dataset, namely the TestCloudIDS dataset, is proposed, which contains fifteen variants of DDoS attacks in the cloud environment. In contrast to other datasets lacking realism and coverage of the latest attack strategies, it closely resembles the real world because of its careful construction. It integrates a wide range of attack situations, utilizing both conventional and current vectors, focusing on incorporating state-of-the-art techniques such as Raven Storm. In addition, we propose “SparkShield”, a technique for intrusion detection using Apache Spark within a big data environment. The effectiveness of “SparkShield” is evaluated through in-depth research using a variety of datasets and simulated attack scenarios. Three existing datasets are used to measure performance: UNSW-NB15, NSL-KDD, CICIDS2017, and the proposed TestCloudIDS dataset. The overall performance of the proposed approach achieved better threat classification and trained with recent attack patterns using the TestCloudIDS dataset.