SMFS-GAN: Style-Guided Multi-class Freehand Sketch-to-Image Synthesis

Abstract

Freehand sketch-to-image (S2I) is a challenging task due to the individualized lines and the random shape of freehand sketches. The multi-class freehand sketch-to-image synthesis task, in turn, presents new challenges for this research area. This task requires not only the consideration of the problems posed by freehand sketches but also the analysis of multi-class domain differences in the conditions of a single model. However, existing methods often have difficulty learning domain differences between multiple classes, and cannot generate controllable and appropriate textures while maintaining shape stability. In this paper, we propose a style-guided multi-class freehand sketch-to-image synthesis model, SMFS-GAN, which can be trained using only unpaired data. To this end, we introduce a contrast-based style encoder that optimizes the network's perception of domain disparities by explicitly modelling the differences between classes and thus extracting style information across domains. Further, to optimize the fine-grained texture of the generated results and the shape consistency with freehand sketches, we propose a local texture refinement discriminator and a Shape Constraint Module, respectively. In addition, to address the imbalance of data classes in the QMUL-Sketch dataset, we add 6K images by drawing manually and obtain QMUL-Sketch+ dataset. Extensive experiments on SketchyCOCO Object dataset, QMUL-Sketch+ dataset and Pseudosketches dataset demonstrate the effectiveness as well as the superiority of our proposed method.