SBoCF: A deep learning-based sequential bag of convolutional features for human behavior quantification

Abstract

The current methods for behavioral quantification heavily rely on manual annotation, which poses a significant challenge due to its labor-intensive and time-consuming nature. This reliance has become a bottleneck, particularly in the context of diagnosing Autism Spectrum Disorder (ASD), where early diagnosis and intervention are crucial for improving patient outcomes. One key area in ASD research is the assessment of atypical hand movements, which are frequently observed in individuals with ASD. To address the limitations of manual annotation, this paper proposes a deep learning-based method for automatically quantifying human behavior, focusing on hand motion evaluation. Specifically, we introduce a Sequential Bag of Convolutional Features (SBoCF) framework that combines the Bag of Words (BoW) approach with a customized skeleton-based CNN gesture classification model. This method allows for the automatic conversion of high-dimensional motion features into discrete behavior sequences, facilitating quantitative hand motor assessment based on established psychological research methods for hand behavior evaluation. Experiments using the DHG-14 dataset have shown promising results, demonstrating the potential of this method to replace traditional time-consuming manual video encoding processes.