A multi-frame fusion video deraining neural network based on depth and luminance features

Abstract

Rainy weather is a common natural occurrence, but capturing clear and accurate images in rainy conditions can be challenging. Consequently, addressing the effects of rain on video imagery is essential for enhancing visual quality and the accuracy and reliability of computer vision applications. In this work, we introduce a novel rain model accompanied by a multi-frame fusion video rain removal algorithm that exploits depth and luminance features. We present an innovative three-stage video deraining methodology that synergizes multi-frame fusion with advanced neural network design to achieve highly realistic and clear rain removal. Our Option-Flow-Depth-Luminance (OFDL) Derain algorithm propels the research forward by offering a new perspective on rain removal. Incorporating depth and luminance features into image processing, we devise a novel rain model that allows the algorithm to handle complex environments and a variety of rain patterns effectively. Additionally, our model takes into account degradation factors such as rain streaks, accumulation, runoff, and occlusion, leading to the generation of more authentic rain-affected images that improve the training and performance evaluation of the model. Experimental tests conducted on the RainSynLight25, RainSynComplex25, and NTURain datasets demonstrate that our method outperforms current state-of-the-art techniques, achieving increases of 4.3 dB, 2.2 dB, and 2.2 dB in PSNR, and enhancements of 0.062, 0.109, and 0.008 in Structural Similarity Index Measure (SSIM), respectively. Moreover, our approach exhibits superior processing speed, further underscoring its practical advantages.