Fusion of face and visual speech information for identity verification

Abstract

Fusion of multiple biometrie characteristics for identity verification has shown obvious merits in contrast to conventional systems based on unimodal biometric features. In this research, a new multimodal verification method is investigated by integrating face and visual speech information simultaneously. Different from face verification, the proposed scheme takes advantage of lip movement features in visual speech, which can significantly decrease the risk of being cheated by a fake face image. To accomplish the work, a Linearity Preserving Projection (LPP) transform and a Projection Local Spatiotemporal Descriptor (PLSD) are applied in the feature extraction for face and visual speech respectively. In order to combine the multisource biometric features, an Extreme Learning Machine (ELM) based fusion strategy is performed on the matching score level to generate a fused score for the final verification. Experiments conducted on the OuluVS database have shown that our proposed method can achieve very satisfying results.