Self-refined variational transformer for image-conditioned layout generation

Layout generation is an emerging computer vision task that incorporates the challenges of object localization and aesthetic evaluation, widely used in advertisements, posters, and slides design. An ideal layout should consider both the intra-domain relationship within layout elements and the inter-domain relationship between layout elements and the image. However, most previous methods simply focus on image-content-agnostic layout generation without leveraging the complex visual information from the image. To address this limitation, we propose a novel paradigm called image-conditioned layout generation, which aims to add text overlays to an image in a semantically coherent manner. Specifically, we introduce the Image-Conditioned Variational Transformer (ICVT) that autoregressively generates diverse layouts in an image. Firstly, the self-attention mechanism is adopted to model the contextual relationship within layout elements, while the cross-attention mechanism is used to fuse the visual information of conditional images. Subsequently, we take them as building blocks of the conditional variational autoencoder (CVAE), which demonstrates attractive diversity. Secondly, to alleviate the gap between the layout elements domain and the visual domain, we design a Geometry Alignment module, in which the geometric information of the image is aligned with the layout representation. Thirdly, we present a self-refinement mechanism to automatically refine the failure case of generated layout, effectively improving the quality of generation. Experimental results show that our model can adaptively generate layouts in the non-intrusive area of the image, resulting in a harmonious layout design.