On the Design of the MITLL Trimodal Dataset for Identity Verification

Abstract

The recent advances in deep learning have led to an increased interest in the development of techniques for multimodal identity verification applications, particularly in the area of biometric fusion. Associated with these efforts is a corresponding need for large scale multimodal datasets to provide the bases for establishing performance baselines for proposed approaches. After examining the characteristics of existing multimodal datasets, this paper will describe the development of the MITLL Trimodal dataset, a new triple-modality collection of data comprising parallel samples of audio, image, and text for 553 subjects. The dataset is formed from YouTube videos and Twitter tweets. Baseline single modality results using a common processing pipeline are presented, along with the results of applying a conventional fusion algorithm to the individual stream scores.