Unsupervised clustering of file dialects according to monotonic decompositions of mixtures

Abstract

This paper proposes an unsupervised classification method that partitions a set of files into non-overlapping dialects based upon their behaviors, determined by messages produced by a collection of programs that consume them. The pattern of messages can be used as the signature of a particular kind of behavior, with the understanding that some messages are likely to co-occur, while others are not. We propose a novel definition for a file format dialect, based upon these behavioral signatures. A dialect defines a subset of the possible messages, called the required messages. Once files are conditioned upon a dialect and its required messages, the remaining messages are statistically independent. With this definition in hand, we present a greedy algorithm that deduces candidate dialects from a dataset consisting of a matrix of file-message data, demonstrate its performance on several file formats, and prove conditions under which it is optimal. We show that an analyst needs to consider fewer dialects than distinct message patterns, which reduces their cognitive load when studying a complex format.