Aggregated Spatio-temporal MLP-Mixer for Violence Recognition in Video Clips

Abstract

Existing violent behavior datasets are not perfect in quantity and quality due to the difficulty of collecting. Although the state-of-the-art Transformer models had shown their capability in behavior recognition, it is unsuitable for the task of short-term behavior understanding (e.g., violent behavior recognition) due to the need for a large amount of data to achieve their best performance. Recently, a simple deep learning architecture, an all multilayer perceptron (MLP) architecture called MLP-Mixer, was proposed against Transformer in the task of a few-sample dataset to obtain competitive results. Motivated by spatio-temporal features on neurons, we invent a dual-form dataset for MLP-Mixer-based model training called aggregated spatio-temporal MLP-Mixer (ASM) to handle video understanding tasks. We show that ASM outperforms the state-of-the-art Transformer models as well as some of the best-performed convolutional neural network (CNN) approaches on three public datasets, smart-city CCTV violence detection dataset (SCVD), real-life violence situations (RLVS) dataset, and Hockey fight. Experimental results further validate our idea on short-term behavior scene understanding improvement.