Comparative Evaluation of Anomaly-Based Controller Area Network IDS

The vulnerability of in-vehicle networks, particularly those based on the Controller Area Network (CAN) protocol, has prompted the development of numerous techniques for intrusion detection on the CAN bus. However, these CAN IDS are often evaluated in disparate experimental settings, with different datasets and evaluation metrics, which hinder direct comparison. This has given rise to efforts at benchmarking and comparative evaluation of CAN IDS under similar experimental conditions to provide an understanding of the relative performance of these CAN IDS. This work contributes to these efforts by reporting results of the comparative evaluation of four statistical and two machine learning-based CAN intrusion detection algorithm, against the Real ORNL Automotive Dynamometer (ROAD) CAN intrusion dataset. The ROAD dataset differs from datasets used in previous work in that it includes the stealthiest possible version of targeted ID fabrication attacks which are more difficult to detect. It also consists of masquerade attacks, which have not been commonly used in comparative evaluation studies. Furthermore, in addition to metrics such as accuracy, precision, recall, and F1-score, we report balanced accuracy, informedness, markedness, and Matthews correlation coefficient, which place equal important on positive and negative classes and are better measures of detection capability, especially for imbalanced datasets. We also report training and testing times for each CAN IDS as an indicator of real-time intrusion detection performance. Results of experiments were found to be generally concordant with previous work, in terms of accuracy, precision, recall, and F1-score. Entropy- and frequency-based CAN IDS were found to be relatively better at detecting attacks, particularly fabrication attacks; while other algorithms did not perform well, as indicated by low MCC scores.