Open set identification of malicious encrypted traffic based on multi-feature fusion

Abstract

In the current network environment, an increasing amount of malicious traffic is transmitted through encrypted channels, carrying control commands and data. With the continuous development of communication protocols and applications, new types of malicious encrypted traffic are emerging, posing significant challenges for network management (e.g., traffic engineering). Therefore, accurately identifying malicious traffic in complex open network spaces has become a hot research topic in network security. In this study, we draw inspiration from channel theory in image science and innovatively convert traffic data into Red-Green-Blue (RGB) image format to achieve the fusion of multiple features. Inspired by image recognition technologies, we have designed a multi-granularity network model that integrates both global and local features, serving as our core network architecture. At the top of the model, we have equipped each known category with a unique autoencoder, using its generated manifold to replace traditional prototypes for model construction. Classification is accomplished through a scoring mechanism that evaluates category membership and by setting thresholds to achieve open set recognition of unknown categories. Relying on our self-created dataset,Malicious and Encrypted Traffic 2024 (MNET2024), we conduct a series of extensive experiments. The results demonstrate that our proposed method exhibits outstanding performance in both closed-set and open-set recognition tasks.