Malware family identification with BIRCH clustering

Abstract

Identifying families of malware is today considered a fundamental problem in the context of computer security. The correct mapping of a malicious sample to a known family simplifies its analysis and allows experts to focus their efforts only on those samples presenting unknown characteristics or behaviours, thus improving the efficiency of the malware analysis process. Grouping malware in families is an activity that can be performed using widely different approaches, but that currently lacks a globally accepted ground truth to be used for comparison. This problem stems from the absence of a formal definition of what a malware family is. As a consequence, in the last few years researchers proposed different methodologies to group a dataset of malicious samples in families. Notable examples include solutions combining labels of commercial anti-malware software, where possible disagreements are solved by majority voting (e.g., AVclass), and dedicated solutions based on machine learning algorithms (e.g., Malheur). In this paper we first present an evaluation to assess the quality of two distinct malware family ground truth datasets. Both include the same set of malware, but one has labels produced by AVclass while the other is based on the clusters identified by Malheur. Then we propose a novel solution for identifying families of similar samples starting from an unlabelled dataset of malware. We leverage features extracted through both static and dynamic analysis, and cluster samples using the BIRCH clustering algorithm. The paper includes an experimental evaluation which shows that BIRCH fits well in the context of malware family identification. Indeed, we prove that BIRCH can be tuned to obtain an accuracy higher than or comparable to standard clustering algorithms, using the ground truths based on AVclass and Malheur. Furthermore, we provide a performance comparison where BIRCH stands out for the low clustering time it provides.