VDA: Deep Learning based Visual Data Analysis in Integrated Edge to Cloud Computing Environment

Abstract

In recent years, video surveillance technology has become pervasive in every sphere. The manual generation of videos’ descriptions requires enormous time and labor, and sometimes essential aspects of videos are overlooked in human summaries. The present work is an attempt towards the automated description generation of Surveillance Video. The proposed method consists of the extraction of key-frames from a surveillance video, objects detection in the key-frames, natural language (English) description generation of the key-frames, and summarizing the descriptions. The key-frames are identified based on a structural similarity index measure. Object detection in a key-frame is performed using the architecture of Single Shot Detection. We used Long Short Term Memory (LSTM) to generate captions from frames. Translation Error Rate (TER) is used to identify and remove duplicate event descriptions. Term frequency-inverse document frequency (TF-IDF) is used to rank the event descriptions generated from a video, and the top-ranked the description is returned as the system generated a summary of the video. We evaluated the Microsoft Video Description Corpus (MSVD) data set to validate our proposed approach, and the system produces a Bilingual Evaluation Understudy (BLEU) score of 46.83.