Knowledge Graphs for Semantic-Aware Anomaly Detection in Video

Video understanding, surveillance and analytics fields have been dynamically expanding over the recent years due to the enormous amount of CCTV, dashcams and phone cameras which generate video data stored on cloud servers, in social networks, in public and private repositories. The video data has a great potential to be used for improving situation awareness, prediction and prevention of unwanted events and disasters in various settings. Still, there is a significant need for methods and ways to understand the large amount of video recordings and to extract hidden patterns and knowledge. Deep learning networks have been successfully applied for video object and anomaly detection tasks. However, while neural networks focus on utilizing features within an object to be detected, the vast amount of background knowledge remains unnoticed. We propose a semantics centered method for video anomaly detection which allows to identify entities that are inconsistent with the scene and thus can be marked as a potential anomaly. Our method is inspired with the way humans comprehend the surroundings with incorporating external knowledge and previous experience. As a source of external knowledge for deep learning networks we maintain a knowledge graph which allows to compute semantic similarity between the detected objects. Similarity of the entities in the frame depends on the distance between the graph vertices which represent the recognized entities. The object which is semantically distinct from other entities in the video is an anomalous one. We conduct experiments on real-life data to empirically prove the efficiency of our approach and provide an enhanced framework that leads to anomaly detection in video with higher accuracy and better interpretability.