A novel malware classification and augmentation model based on convolutional neural network

The rapid development and widespread use of the Internet have led to an increase in the number and variety of malware proliferating via the Internet. Malware is the general nomenclature for malicious software. Malware classification is an undecidable problem and technically NP hard problem because the halting problem is NP hard. In this study, we proposed a convolutional neural network based novel method for malware classification. Since CNN models use the images as input, bytes files are transformed to gray separately and RGB image formats for the classification process. A new approach called B2IMG is developed for the transformation of bytes file. Moreover, a new CycleGAN-based data augmentation method is proposed to address the problem of imbalanced data size between malware families. The proposed system was tested on the BIG2015, and DumpWare10 datasets. According to the experimental results, classification performance increased thanks to the proposed data augmentation method. The accuracy of the classification is 99.86% for the BIG2015 dataset and 99.60% for the dataset.