A novel framework for diverse video generation from a single video using frame-conditioned denoising diffusion probabilistic model and ConvNeXt-V2

Abstract

The Denoising Diffusion Probabilistic Model (DDPM) has significantly advanced video generation and synthesis. DDPM relies on extensive datasets for its training process. The study presents a novel method for generating videos from a single video through a frame-conditioned Denoising Diffusion Probabilistic Model (DDPM). Additionally, incorporating the ConvNeXt-V2 model significantly boosts the framework’s feature extraction, improving video generation performance. Addressing the data scarcity challenge in video generation, the proposed model framework exploits a single video’s intrinsic complexities and temporal dynamics to generate diverse and realistic sequences. The model’s ability to generalize motion is demonstrated through thorough quantitative assessments, wherein it is trained on segments of the original video and evaluated on previously unseen frames. Integrating Global Response Normalization and Sigmoid-Weighted Linear Unit (SiLU) activation functions within the DDPM framework has enhanced generated video quality. Comparatively, the proposed model markedly outperforms the Sinfusion model across crucial image quality metrics, achieving a lower Freschet Video Distance (FVD) score of 106.52, lower Learned Perceptual Image Patch Similarity (LPIPS) score of 0.085, higher Structural Similarity Index Measure (SSIM) score of 0.089, higher Nearest-Neighbor-Field (NNF) based diversity (NNFDIV) score of 0.44. Furthermore, the model achieves a Peak Signal to Noise Ratio score of 23.95, demonstrating its strength in preserving image integrity despite noise. The integration of Global Response Normalization and SiLU significantly enhances content synthesis, while ConvNeXt-V2 boosts feature extraction, amplifying the model’s efficacy.