Write What You Want: Applying Text-to-video Retrieval to Audiovisual Archives

Abstract

Audiovisual (AV) archives, as an essential reservoir of our cultural assets, are suffering from the issue of accessibility. The complex nature of the medium itself made processing and interaction an open challenge still in the field of computer vision, multimodal learning, and human-computer interaction, as well as in culture and heritage. In recent years, with the raising of video retrieval tasks, methods in retrieving video content with natural language (text-to-video retrieval) gained quite some attention and have reached a performance level where real-world application is on the horizon. Appealing as it may sound, such methods focus on retrieving videos using plain visual-focused descriptions of what has happened in the video and finding videos such as instructions. It is too early to say such methods would be the new paradigms for accessing and encoding complex video content into high-dimensional data, but they are indeed innovative attempts and foundations to build future exploratory interfaces for AV archives (e.g. allow users to write stories and retrieve related snippets in the archive, or encoding video content at high-level for visualisation). This work filled the application gap by examining such text-to-video retrieval methods from an implementation point of view and proposed and verified a classifier-enhanced workflow to allow better results when dealing with in-situ queries that might have been different from the training dataset. Such a workflow is then applied to the real-world archive from Télévision Suisse Romande (RTS) to create a demo. At last, a human-centred evaluation is conducted to understand whether the text-to-video retrieval methods improve the overall experience of accessing AV archives.