Self-organizing resilient network sensing (SornS) with very large scale anomaly detection

Abstract

Anomaly detection promises to find elements of abnormality in a field of data. Computational barriers constrain anomaly detection to sparse subsets of total anomaly space. Barriers manifest in three ways — conserving both pattern memory capacity and pattern matching cycle time, while closing off scalability. The research reported here has discovered and analyzed a technology to eliminate two of these barriers, memory capacity and cycle time, and by targeting implementation at a new VLSI pattern processor, eliminate the third scalability barrier. An example shows how 10 to the 15 patterns integrated as a single gang detector can be stored in 193 bytes of memory, with much larger pattern magnitudes practical as well. The architecture of the gang detector enables complete processing of all 10 to the 15 patterns in time determined by the number of features in a single pattern, rather than the total number of patterns. Scalability is provided by a reconfigurable massively parallel VLSI pattern-matching processor chip that can accommodate a virtually unbounded number of such gang detectors. Anomalous behavior detection promises a way round the limitations of looking only for known attack patterns, but it raises new issues in the cyber domain of higher false positive rates and questionable normal-behavior stability. Work reported in this paper describes the nature and capability of gang detector employment, and suggests that the traditional issues of anomaly detection can be addressed with an architecture that engages in continuous learning and re-profiling of normal behavior, and employs a sensemaking hierarchy to reduce false positives. The architecture is based on process patterns from the biological immune system combined with process patterns of mammalian cortical hierarchical sensemaking.