ShadowGAN-Former: Reweighting self-attention based on mask for shadow removal

Abstract

Shadow removal remains a challenging visual task aimed at restoring the original brightness of shadow regions in images. Many existing methods overlook the implicit clues within non-shadow regions, leading to inconsistencies in the color, texture, and illumination of the reconstructed shadow-free images. To address these issues, we propose an efficient hybrid model of Transformer and Generative Adversarial Network (GAN), named ShadowGAN-Former, which utilizes information from non-shadow regions to assist in shadow removal. We introduce the Multi-Head Transposed Attention (MHTA) and Gated Feed-Forward Network (Gated FFN), designed to enhance focus on key features while reducing computational costs. Furthermore, we propose the Shadow Attention Reweight Module (SARM) to reweight the self-attention maps based on the correlation between shadow and non-shadow regions, thereby emphasizing the contextual relevance between them. Experimental results on the ISTD and SRD datasets show that our method outperforms popular and state-of-the-art shadow removal algorithms, with the SARM module improving PSNR by 5.42% and reducing RMSE by 14.76%.