Controllable image generation based on causal representation learning

IF 2.7 3区工程技术 Q2 COMPUTER SCIENCE, INFORMATION SYSTEMS Frontiers of Information Technology & Electronic Engineering Pub Date : 2024-01-01 DOI:10.1631/fitee.2300303

{"title":"Controllable image generation based on causal representation learning","authors":"","doi":"10.1631/fitee.2300303","DOIUrl":null,"url":null,"abstract":"<h3>Abstract</h3> <p>Artificial intelligence generated content (AIGC) has emerged as an indispensable tool for producing large-scale content in various forms, such as images, thanks to the significant role that AI plays in imitation and production. However, interpretability and controllability remain challenges. Existing AI methods often face challenges in producing images that are both flexible and controllable while considering causal relationships within the images. To address this issue, we have developed a novel method for causal controllable image generation (CCIG) that combines causal representation learning with bi-directional generative adversarial networks (GANs). This approach enables humans to control image attributes while considering the rationality and interpretability of the generated images and also allows for the generation of counterfactual images. The key of our approach, CCIG, lies in the use of a causal structure learning module to learn the causal relationships between image attributes and joint optimization with the encoder, generator, and joint discriminator in the image generation module. By doing so, we can learn causal representations in image’s latent space and use causal intervention operations to control image generation. We conduct extensive experiments on a real-world dataset, CelebA. The experimental results illustrate the effectiveness of CCIG.</p>","PeriodicalId":12608,"journal":{"name":"Frontiers of Information Technology & Electronic Engineering","volume":"14 1","pages":""},"PeriodicalIF":2.7000,"publicationDate":"2024-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Frontiers of Information Technology & Electronic Engineering","FirstCategoryId":"5","ListUrlMain":"https://doi.org/10.1631/fitee.2300303","RegionNum":3,"RegionCategory":"工程技术","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q2","JCRName":"COMPUTER SCIENCE, INFORMATION SYSTEMS","Score":null,"Total":0}

引用次数: 0

Abstract

Artificial intelligence generated content (AIGC) has emerged as an indispensable tool for producing large-scale content in various forms, such as images, thanks to the significant role that AI plays in imitation and production. However, interpretability and controllability remain challenges. Existing AI methods often face challenges in producing images that are both flexible and controllable while considering causal relationships within the images. To address this issue, we have developed a novel method for causal controllable image generation (CCIG) that combines causal representation learning with bi-directional generative adversarial networks (GANs). This approach enables humans to control image attributes while considering the rationality and interpretability of the generated images and also allows for the generation of counterfactual images. The key of our approach, CCIG, lies in the use of a causal structure learning module to learn the causal relationships between image attributes and joint optimization with the encoder, generator, and joint discriminator in the image generation module. By doing so, we can learn causal representations in image’s latent space and use causal intervention operations to control image generation. We conduct extensive experiments on a real-world dataset, CelebA. The experimental results illustrate the effectiveness of CCIG.

查看原文

微信好友朋友圈 QQ好友复制链接

本刊更多论文

基于因果表征学习的可控图像生成

摘要人工智能生成内容（AIGC）在模仿和制作方面发挥着重要作用，因此已成为制作图像等各种形式的大规模内容不可或缺的工具。然而，可解释性和可控性仍然是一项挑战。现有的人工智能方法在制作既灵活又可控的图像，同时还要考虑图像内部的因果关系方面往往面临挑战。为了解决这个问题，我们开发了一种新颖的因果可控图像生成（CCIG）方法，它将因果表征学习与双向生成对抗网络（GANs）相结合。这种方法能让人类控制图像属性，同时考虑到生成图像的合理性和可解释性，还能生成反事实图像。我们的方法（CCIG）的关键在于使用因果结构学习模块来学习图像属性之间的因果关系，并与图像生成模块中的编码器、生成器和联合判别器进行联合优化。通过这种方法，我们可以学习图像潜在空间中的因果表征，并使用因果干预操作来控制图像生成。我们在真实世界数据集 CelebA 上进行了大量实验。实验结果证明了 CCIG 的有效性。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文去求助

来源期刊

Frontiers of Information Technology & Electronic Engineering COMPUTER SCIENCE, INFORMATION SYSTEMSCOMPU-COMPUTER SCIENCE, SOFTWARE ENGINEERING

CiteScore

6.00

自引率

10.00%

发文量

1372

期刊介绍： Frontiers of Information Technology & Electronic Engineering (ISSN 2095-9184, monthly), formerly known as Journal of Zhejiang University SCIENCE C (Computers & Electronics) (2010-2014), is an international peer-reviewed journal launched by Chinese Academy of Engineering (CAE) and Zhejiang University, co-published by Springer & Zhejiang University Press. FITEE is aimed to publish the latest implementation of applications, principles, and algorithms in the broad area of Electrical and Electronic Engineering, including but not limited to Computer Science, Information Sciences, Control, Automation, Telecommunications. There are different types of articles for your choice, including research articles, review articles, science letters, perspective, new technical notes and methods, etc.