Adapting visible-light-image diffusion model for infrared image restoration in rainy weather

Abstract

Infrared images captured in rainy conditions always suffer from significant quality degradation, limiting the utilization of infrared equipment in rainy weather. However, the problems mentioned above have not been effectively solved yet. On the one hand, no research has been devoted to developing methods for rainy weather infrared image restoration. On the other hand, there is no available paired infrared image restoration dataset for training. To tackle the aforementioned issues, we propose a novel framework, named InfDiff, to restore low-quality infrared images via High-Quality Visible-light image Prior. Meanwhile, we establish a realistic paired infrared rainy weather dataset for model training. Specifically, the proposed InfDiff consists of an Infrared Restoration Transformer and a Prior Generation Module. InfRestormer achieves degradation removal by modeling the inverse process of infrared degradation generating and can efficiently improve image quality using High-Quality Infrared image Prior. Correspondingly, the Prior Generation Module generates High-Quality Visible-light image Prior employing a diffusion model pre-trained on abundant visible-light images, and converts it into High-Quality Infrared image Prior via adapter fine-tuning for exploitation by InfRestormer. The above approach allows employing abundant visible-light data to effectively improve the quality of infrared images with the limited amount and diversity of infrared training data. In addition, to train the InfRestormer and fine-tune the adapter, we propose a realistic degradation simulation scheme and synthesize a paired clean-degraded infrared image dataset for the first time. In summary, we find that information in high-quality visible-light images can help restore corrupted content in low-quality infrared images. Based on the above finding, we propose the first rainy weather infrared image restoration framework, named InfDiff. Additionally, we synthesized the first rainy weather infrared image restoration dataset for model training. Extensive experiments demonstrate that our method significantly outperforms the existing image restoration scheme.