Detecting malicious DoH traffic: Leveraging small sample analysis and adversarial networks for detection

In light of the escalating frequency of DNS attacks, it is imperative to bolster user security and privacy through the encryption of DNS queries. However, conventional methods for detecting DNS traffic are no longer effective in identifying encrypted traffic, particularly with the utilization of the DNS-over-HTTPS (DoH) protocol, which employs secure HTTPS for DNS resolution. To confront this challenge, we propose a novel model for detecting malicious DoH traffic, named DoH-TriCGAN, which distinguishes between non-DoH, benign DoH, and malicious DoH traffic. DoH-TriCGAN employs a conditional generative adversarial network comprising three network components, for which we only provide additional information to the generator. We extracted different small sample datasets and large sample dataset from the CIRA-CIC-DoHBrw-2020 dataset, to evaluate the efficiency and effectiveness of the proposed DoH-TriCGAN model, and compared the quality of the generated synthetic data. To establish a benchmark, we utilized the six metrics – accuracy, precision, recall, F1-score, ROC_AUC, and PR_AUC – to assess the performance of our model. The results demonstrate our proposed model outperforms the other five models (RF, XGBoost, BiGRU, Autoencoder, Transformer), showing the best performance particularly in scenarios with limited training samples, while also demonstrating data expansion capabilities by generating high-quality synthetic data to address the issue of insufficient network traffic.