Low-Complexity Video Classification using Recurrent Neural Networks

Deep learning has led to great successes in computer vision tasks such as image classification. This is mostly attributed to the availability of large image datasets such as ImageNet. However, the progress in video classification has been slower, especially due to the small size of available video datasets and larger computational and memory demands. To promote innovation and advancement in this field, Google announced the YouTube-8M dataset in 2016, which is a public video dataset containing about 8-million tagged videos. In this paper, we train several deep neural networks for video classification on a subset of YouTube-8M. Our approach is based on extracting frame-level features using the Inception-v3 network, which are later used by recurrent neural networks with LSTM/BiLSTM units for video classification. We focus on network architectures with low computational requirements and present a detailed performance comparison. We show that for 5 categories, more than 96% of the videos are labeled correctly, where for 10 categories more than 89% of the videos are labeled correctly. We demonstrate that transfer learning leads to substantial saving in training time, while offering good results.