Relevance feature selection with data cleaning for intrusion detection system

Abstract

Labeled datasets play a major role in the process of validating and evaluating machine learning techniques in intrusion detection systems. In order to obtain good accuracy in the evaluation, very large datasets should be considered. Intrusion traffic and normal traffic are in general dependent on a large number of network characteristics called features. However not all of these features contribute to the traffic characteristics. Therefore, eliminating the non-contributing features from the datasets, to facilitate speed and accuracy to the evaluation of machine learning techniques, becomes an important requirement. In this paper we suggest an approach which analyzes the intrusion datasets, evaluates the features for its relevance to a specific attack, determines the level of contribution of feature, and eliminates it from the dataset automatically. We adopt the Rough Set Theory (RST) based approach and select relevance features using multidimensional scatter-plot automatically. A pair-wise feature selection process is adopted to simplify. In our previous research we used KDD'99 dataset and validated the RST based approach. There are lots of redundant data entries in KDD'99 and thus the machine learning techniques are biased towards most occurring events. This property leads the algorithms to ignore less frequent events which can be more harmful than most occurring events. False positives are another important drawback in KDD'99 dataset. In this paper, we adopt NSL-KDD dataset (an improved version of KDD'99 dataset) and validate the automated RST based approach. The approach presented in this paper leads to a selection of most relevance features and we expect that the intrusion detection research using KDD'99-based datasets will benefit from the good understanding of network features and their influences to attacks.