ZMAR-SNFlow:Restoration for low-light images with massive zero-element pixels

Abstract

Under real-world extremely low-light conditions, many low-light RGB (Red, Green, Blue) images contains massive zero-element pixels (a zero-element pixel is defined as that a color pixel with three RGB values contain no less than one zero). Low-light images with massive zero-element pixels suffer both light weakness and information loss. Existing low-light image enhancement methods aim to amplify the low-light, whereas seldomly consider to restore the information loss caused by massive zero-element pixels. To tackle above issue, firstly, we construct a zero-element mask set that contains many zero-element masks from real-world extremely low-light night traffic monitoring (NTM) images. Each zero-element mask is a binary image, where 1 and 0 are corresponding to zero-element pixels and other pixels. Secondly, we propose a novel flow-based generative method ZMAR-SNFlow to restore low-light images with massive zero-element pixels. ZMAR-SNFlow consists of a zero-element mask attention based Restormer (ZMAR) encoder and a strengthened normalizing flow (SNFlow). Specifically, we proposed a zero-element mask attention (ZMA) module, which is combined with the Restormer module to form the ZMAR module, and ZMAR is used to develop the ZMAR encoder. Then, we propose to insert the unconditional affine coupling layer into the flow step of existing normalizing flow to form SNFlow. ZMAR-SNFlow learns to map the output of SNFlow into a standard normal distribution, and the inverse network of SNFlow takes the latent features of the low-light image as its input to infer the enhanced image. Finally, experimental results on benchmark datasets show that the proposed ZMAR-SNFlow can achieve state-of-the-art (SOTA) performance for low-light images with massive zero-element pixels. The source code and pre-trained models are available at https://github.com/NJUPT-IPR-ZhangBo/ZMAR-SNFlow.