Beyond known threats: A novel strategy for isolating and detecting unknown malicious traffic

Traditional network intrusion detection systems excel at screening known attack types, but face significant challenges when dealing with unseen malicious traffic, often misclassifying such novel attacks into known classes. Existing unknown malicious traffic detection methods frequently fail to effectively control the distribution of known classes in the representation space and do not reserve sufficient representation space for unknown malicious traffic, blurring the boundaries between known and unknown traffic classifications. Furthermore, because known traffic types are centrally distributed within the representation space, whereas unknown malicious traffic types are scattered throughout, additional constraint processing of hard samples is required. To this end, we propose a one-class classification model for unknown malicious traffic called OC-MAL. The core of OC-MAL is to make full use of hard samples to force constraints on the distribution of the known classes in the representation space, separating the unknown and known classes well and realizing the accurate detection of unknown malicious traffic. We fuse a Deep SVDD and an autoencoder in which the reconstruction loss ensures that the latent variables of known classes retain rich category information and the distance loss ensures that known classes are tightly clustered at the center of a hypersphere in representation space. Moreover, the two are combined to further improve the discriminative power on unknown malicious traffic. We evaluated the OC-MAL model on a public malicious traffic dataset. The results showed that it achieves an average AUC value of 95.16% on the malicious traffic dataset, outperforming other state-of-the-art methods.