SynGauss: Real-Time 3D Gaussian Splatting for Audio-Driven Talking Head Synthesis

Abstract

In the field of virtual human generation, Neural Radiance Fields (NeRF) have made significant strides in precise geometric modeling and color accuracy, establishing new benchmarks for complex viewpoint synthesis and 3D reconstruction. Despite these advancements, existing methods face substantial limitations in real-time dynamic facial expression capture and managing high-frequency details, particularly in rapid facial movements and accurate lip synchronization. These constraints are largely due to the high computational load and the dense data requirements hamper real-time rendering. Additionally, traditional radiance fields struggle to capture subtle facial changes driven by audio, often resulting in animations that lack expressiveness and naturalness. Building upon the foundation laid by TalkingGaussian,this paper introduces an advanced framework named SynGauss that employs 3D Gaussian Splatting to precisely decouple facial and lip movements. We have enhanced this approach by incorporating lip expression coefficients and a regional multi-head attention mechanism, which allow for detailed and controlled animation of complex facial dynamics. Our modifications provide a more refined control over lip movements and facial expressions, significantly improving the realism and expressiveness of the animations while maintaining the efficiency required for real-time applications. This approach holds great promise for real-time applications such as virtual assistants and immersive entertainment experiences, offering more realistic and controllable animation generation.(Project address https://github.com/zzyfight0703/SynGauss/tree/main)