Spectral malware behavior clustering

Abstract

We develop a version of spectral clustering and empirically study its performance when applied to behavior-based malware clustering. In 2011, a behavior-based malware clustering algorithm was reported by Rieck et al. We hypothesize that, owing to the more complex nature of our algorithm, it will exhibit higher accuracy than Rieck's but will require greater run-time. Through experiments using three different malware datasets, we largely substantiate this hypothesis. Our approach had comparable or superior accuracy to Rieck's over all of its parameter settings examined and ours had higher run-times (nonetheless, ours had run-times of less than one minute on all datasets). We also found our algorithm had no clear accuracy advantage, but much smaller run-times than Hierarchical Agglomerative Clustering.