Face Recognition Using Convolutional Neural Networks and Metadata in a Feature Fusion Model

Recent advances in science and technology are raising ever-increasing security issues. In response, traditional authentication systems based on knowledge or possession have been developed, but these soon came up against limitations in terms of security and practicality. To overcome these limitations, other systems based on the individual's unique characteristics, known as biometric modalities, were developed. Of the various ways of improving the performance of biometric systems, feature fusion and the joint use of a pure biometric modality and a soft biometric modality (multi-origin biometrics) are highly promising. Unfortunately, however, we note a virtual absence of multi-origin systems in a feature fusion strategy. For our work, we therefore set out to design such a multi-origin system fusing facial features and skin color. Using OpenCV (Open Computer Vision) and Python, we extracted facial features and merged them with skin color to characterize each individual. The HOG (Histogram of Oriented Gradients) algorithm was used for face detection, and Google's deep neural network for encoding. For skin color, segmentation in the HSV (Hue, Saturation, Value) color space enabled us to isolate the skin in each image, and thanks to the k-means algorithm we had detected the dominant skin colors. The system designed in this way enabled us to go from 81.8% as a TR (Recognition Rate) with the face alone to 86.8% after fusion for a TFA (False Acceptance Rate) set at 0.1% and from 0.6% as a TEE (Equal Error Rate) to 0.55%.