Few-shot Adversarial Audio Driving Talking Face Generation

Abstract

Talking-face generation is an interesting and challenging problem in computer vision and has become a research focus. This project aims to generate real talking-face video sequences, especially lip synchronization and head motion. In order to create a personalized talking-face model, these works require training on large-scale audio-visual datasets. However, in many practical scenarios, the personalized appearance features, and audio-video synchronization relationships need to be learned from a few lip synchronization sequences. In this paper, we consider it as a few-shot image synchronization problem: synthesizing talking-face with audio if there are additionally a few lip-synchronized video sequences as the learning task? We apply the reptile methods to train the meta adversarial networks and this meta-model could be adapted on just a few references sequences and done quickly to learn the personalized references models. With meta-learning on the dataset, the model can learn the initialization parameters. And with few adapt steps on the reference sequences, the model can learn quickly and generate highly realistic images with more facial texture and lip-sync. Experiments on several datasets demonstrated significantly better results obtained by our methods than the state-of-the-art methods in both quantitative and quantitative comparisons.