Outlier detection using semantic sensors

Abstract

We describe a technique that calculates the expected relationships among attributes from training data, and uses this to generate anomaly scores reflecting the intuition that a record with anomalous values for related attributes is more anomalous than one with anomalous values for unrelated attributes. The expected relations among attributes are calculated in two ways: using a data-dependent projection via singular value decomposition, and using the maximal information coefficient. Sufficiently anomalous records are displayed on a sensor dashboard, making it possible for an analyst to judge why each record has been classified as anomalous. The technique is illustrated for an intrusion detection dataset, and a set of contract descriptors.