A Method for Fast Outlier Detection in High Dimensional Database Log

Abstract

An easy to implement and effective outlier detection method is proposed in this paper, which is a two-stage process combining the kd-tree structure and the Isolation Forest (Forest) method. We use kd-tree to split high dimensional data into groups, and then apply Forest to each group to calculate anomaly scores which help to identify outliers. This method is fast since it decides anomaly on groups of a dataset instead of the whole dataset, meanwhile the accuracy is assured by Forest. We tested our method with synthetic and real-world data set to illustrates its application to data base access logs.