Two-tiered Spatio-temporal Feature Extraction for Micro-expression Classification

Abstract

This paper proposed a framework called DAuLiLSTM (DAuVi + LiLSTM) for Micro-expression (ME) classification. It extracts spatio-temporal (ST) features through two novel components: dynamic image of augmented video (DAuVi) and Lightnet with LSTM (LiLSTM). The first component presents a unique strategy to generate multiple dynamic images of each original ME video that contain the relevant ST features. It proposes an algorithm that works as a sliding window and ensures the incorporation of the apex frame in each dynamic image. The second component further processes those images to extract additional ST features. The LiLSTM consists of two deep networks: Lightnet and LSTM. The Lightnet extracts the spatial information and LSTM learns the temporal sequences. A combination of both components extracts ST features sequentially twice and ensures that the model captures all ST features. We found that our model outperforms 14 state-of-the-art techniques in accuracy and F1-score on three ME datasets.