AutoRoC-DBSCAN: automatic tuning of DBSCAN to detect malicious DNS tunnels

Abstract

Modern attacks, such as advanced persistent threats, hide command-and-control channels inside authorized network traffic like DNS or DNS over HTTPS to infiltrate the local network and exfiltrate sensitive data. Detecting such malicious traffic using traditional techniques is cumbersome especially when the traffic encrypted like DNS over HTTPS. Unsupervised machine learning techniques, and more specifically density-based spatial clustering of applications with noise (DBSCAN), can achieve good results in detecting malicious DNS tunnels. However, DBSCAN requires manually tuning two hyperparameters, whose optimal values can differ depending on the dataset. In this article, we propose an improved algorithm called AutoRoC-DBSCAN that can automatically find the best hyperparameters. We evaluated and obtained good results on two different datasets: a dataset we created with malicious DNS tunnels and the CIRA-CIC-DoHBrw-2020 dataset with malicious DoH tunnels.