ESTS-GCN: An Ensemble Spatial–Temporal Skeleton-Based Graph Convolutional Networks for Violence Detection

Abstract

Surveillance systems are essential for social and personal security. However, monitoring multiple video feeds with multiple targets is challenging for human operators. Therefore, automatic and smart surveillance systems have been introduced to support or replace traditional surveillance systems and build safer communities. Advancements in artificial intelligence techniques, particularly in the field of computer vision, have boosted this area of research. Most existing works have focused on image-based (RGB-based) machine learning and deep learning algorithms for detecting anomalous and violent events. In this study, we propose a unique Ensemble Spatial–Temporal Skeleton-Based Graph Convolutional Networks (ESTS-GCNs) model for violence detection that automatically uses spatial and temporal data to detect violence in surveillance videos. Skeleton-based algorithms are less sensitive to pixel-based noise and background interference, making them excellent candidates for activity and anomaly detection. Our proposed ensemble-based architecture utilizes Graph Convolutional Networks (GCNs) and comprises multiple spatial and temporal modules. Three different spatial pipelines are exploited: channel-wise topologies, self-attention mechanism, and graph attention networks. The models were trained and evaluated using two skeleton-based datasets introduced by us: Skeleton-based Real-Life Violence Situations (RLVS) and NTU-Violence (NTU-V). Our model achieved a maximum accuracy of around 93% and outperformed existing models by more than 10%.