Improved Likelihood Ratios for Surveillance Video Face Recognition with Multimodal Feature Pairing

The accuracy of face recognition in real-world surveillance videos plays a crucial role in forensic investigation and security monitoring systems. Despite advancements in technology, face recognition methods can be influenced by variations in pose, illumination, and facial expression that often occur in these videos. To address this issue, we propose a new method for images-to-video face recognition that pairs face images with multiple attributes (soft labels) and face image quality (FIQ). This is followed by the application of three calibration methods to estimate the likelihood ratio, which is a statistical measure commonly used in forensic investigations. To validate the results, we test our method on the ENFSI proficiency test 2015 dataset, using SCFace and ForenFace as calibration datasets and three embedding models: ArcFace, FaceNet, and QMagFace. Our results indicate that using only high quality frames can improve face recognition performance for forensic purposes compared to using all frames. The best results were achieved when using the highest number of common attributes between the reference image and selected frames, or by creating a single common embedding from the selected frames, weighted by the quality of each frame’s face image.