Command Evaluation in Encrypted Remote Sessions

Abstract

Intrusion Detection Systems (IDS) are integral components for the detection of malicious code and attacks. Detection methods can be differentiated in signature-based and anomaly-based systems. While the former ones search for well-known patterns which are available in a database, the latter ones build a model of the normal behavior of a network and later on attacks can be detected by measuring significant deviation of the network status against the normal behavior described by the model. Often this requires the availability of the payload of the network packets. If encryption protocols like SSL or SSH are used, searching for attack signatures in the payload is not possible any longer and also the usage of behavior based techniques is limited: Statistical methods like flow evaluation can be used for anomaly detection, but application level attacks hidden in the encrypted traffic can be undetectable. At the moment, only a few systems are designed to cope with encrypted network traffic. Even so, none of these systems can be easily deployed in general because of the need for protocol modifications, special infrastructures or because of high false alarm rates which are not acceptable in a production environment. In this paper, we propose a new IDS for encrypted traffic which identifies command sequences in encrypted network traffic and evaluates the attack possibility of them. The encrypted traffic is clustered and possibilities for different commands are calculated. Based on that, command sequences are analysed. The system evaluates probabilities for commands and command sequences and the likeliness for an attack based on the identified sequences without a decryption of the packets. Because of only using statistical data gathered from the network traffic, the system can be deployed in general. The current prototype of the system focuses on the command evaluation.