Anomaly Detection on User Terminals Based on Outbound Traffic Filtering by DNS Query Monitoring and Application Program Identification

Malware attacks have become one of the most critical issues in the Internet nowadays. Most types of malware, after infecting a computer, attempt contacts to the Command and Control (C&C) servers using IP addresses or Fully Qualified Domain Name (FQDN) for further instructions. In the former case, the malware connects to the C&C servers directly without DNS name resolutions, while in the later case, DNS name resolutions for obtaining the IP addresses of the C&C servers are required. In both cases, the outbound traffic will be initialized by an unrecognized application program, the malware. In this research, we focus on these peculiarities and propose an anomaly detection system on user terminals based on outbound traffic filtering by using Software Defined Network (SDN) and DNS Response Policy Zone (DNS RPZ) technologies. In the proposed system, the outbound traffic initialized by unrecognized application programs or destined to the IP addresses obtained without DNS name resolutions will be detected and blocked on user terminals. What's more, in order to reduce false positive detections, an alert message will make the users decide whether or not to allow the detected traffic. We implemented a prototype system on MacOS machine and conducted feature evaluations. According to the evaluation results, we confirmed that the proposed system worked exactly as we designed with the features of detection, blocking and alerting anomalies on user terminals.