Image inpainting for periodic discrete density defects via frequency analysis and an adaptive transformer-GAN network

Abstract

Image inpainting based on deep learning has made significant progress in addressing regular and coherent irregular defects. However, little has been studied on periodic discrete density (PDD) defects that are prevalent in microscopic images obtained by advanced instruments like transmission electron microscopes (TEM) and scanning tunneling microscopes (STM). The PDD defects usually introduce low-frequency noise in the fast Fourier transform (FFT) images, preventing the extraction of useful information particularly in the low-frequency regions. Despite its significant impact, no method has been reported to date to efficiently remove the PDD-induced noise from the FFT of high-resolution microscopic images. In this study, we introduced a novel GAN-based two-stage network (FGTNet), a novel coarse-to-fine inpainting framework, which is built upon the architecture of Generative Adversarial Networks (GAN) and transformer blocks. By integrating the information from both frequency and spatial domains, contextual structures are preserved and high-frequency details are generated in our method. We also proposed an adaptive-window transformer block (A-LeWin) to enhance the spatial feature representation and to fully use the information around the defects. To validate our approach, we constructed a specialized microscopic image dataset with 2730 training samples and 105 testing samples. For comparison, we also extended the experiments to the public Describable Texture Dataset (DTD) and coherence defects that are often discussed in the field of image inpainting. The experiment results indicate that our method performs well on six pixel-level and perceptual-level metrics, and shows the best performance and visual effect of coherent texture.