Identifying forensically uninteresting files in a large corpus

For digital forensics, eliminating the uninteresting is often more critical than finding the interesting since there is so much more of it. Published software-file hash values like those of the National Software Reference Library (NSRL) have limited scope. We discuss methods based on analysis of file context using the metadata of a large corpus. Tests were done with an international corpus of 262.7 million files obtained from 4018 drives. For malware investigations, we identify clues to malware in context, and show that using a Bayesian ranking formula on metadata can increase recall by 5.1 while increasing precision by 1.7 times over inspecting executables alone. For more general investigations, we show that using together two of nine criteria for uninteresting files, with exceptions for some special interesting files, can exclude 77.4% of our corpus instead of the 23.8% that were excluded by NSRL. For a test set of 19,784 randomly selected files from our corpus that were manually inspected, false positives after file exclusion (interesting files identified as uninteresting) were 0.18% and false negatives (uninteresting files identified as interesting) were 29.31% using our methods. The generality of the methods was confirmed by separately testing two halves of our corpus. Few of our excluded files were matched in two commercial hash sets. This work provides both new uninteresting hash values and programs for finding more.