MinCloud: Trusted and transferable MinHash-based framework for unknown malware detection for Linux cloud environments

Linux clouds have become an attractive target for cyber-attacks. However, existing detection solutions for Linux clouds have variety of limitations. Some of the solutions are untrusted, incapable of detecting unknown malware, or rely on a human expert to define the features. Other solutions are trusted but require a large amount of computational resources or have a limited ability to detect rootkits, fileless malware, or malware on a different server. In this study, we propose MinCloud, a trusted and transferable MinHash-based framework for unknown malware detection in Linux virtual servers that overcomes the limitations of existing solutions. In the first stage, we acquired volatile memory dumps from virtual servers by querying the hypervisor in a trusted manner and then analyzed them using the MinHash method. Finally, the MinHash characteristics are harnessed by applying machine learning classifiers to achieve precise malware detection. MinCloud was evaluated on widely used Linux virtual servers, various benign and malicious applications, and 23,000 volatile memory dumps, each representing different behaviors of the examined servers and the executed applications over time. MinCloud's evaluation shows it can (1) detect unknown malware, (2) classify unknown malware according to its malware category, (3) detect fileless attacks and rootkit malware, and (4) provide accurately transfer detection between different Linux servers. MinCloud outperformed state-of-the-art trusted detection methods and commonly used antiviruses.