Efficient Video Retrieval Method Based on Transition Detection and Video Metadata Information

Abstract

In this paper, we propose an event retrieval support system that quickly finds videos in a large database based on user-entered content. The system addresses the challenges of providing fast and relevant results for a dataset of over 400 hours of videos and developing user-friendly tools. To achieve fast retrieval, we convert the videos into compact semantic features. This involves two steps: (1) Identifying keyframes that represent different content and (2) Extracting semantic features from these frames. We first use the TransNet model to find transition frames, which split the video into scenes with different content. Then we will extract the keyframes which are evenly distributed in these scenes. Finally, the CLIP model is used to extract features from these keyframes and connect them with text. This forms a compact and semantic feature database. When users search with text, we convert it into features and measure similarity with the database using cosine distance, then the most similar video is retrieved. In cases where CLIP model fails, we recommend leveraging news headlines and audio by applying Optical Character Recognition (OCR) and Automatic Speech Recognition (ASR) on videos to form a text database and comparing the entered text with this text database. Experimental results on a Vietnamese media news dataset demonstrate the effectiveness and accuracy of our method.