The Avatar Facial Expression Reenactment Method in the Metaverse based on Overall-Local Optical-Flow Estimation and Illumination Difference

Abstract

To implement a metaverse exhibition interaction system, the instability problem of high-quality avatar facial reenactment must be considered. How to void identity limitations and eliminate artifacts are key challenges for avatar reenactment. It also lacks the support of the application system it is implemented in. We propose a metaverse system architecture oriented to emotional interaction. And we propose a novel method for avatar expression reenactment named Overall-Local Feature Warping Fusion Model based on Optical-Flow field prediction. We solve the identity limitation by overall optical-flow estimation and local optical-flow estimation and eliminate artifacts by illumination consistency. We compare with the mainstream optical flow face reenactment methods and outperform them in identity similarity, structural similarity, and facial action unit recognition ratio. We experimentally compared our method improves by an average improvement of 3.79%. And we also implement our method in the metaverse exhibition system. Although we satisfy most of the interaction scenarios, our method is still insufficient in some side-face cases.