VToonify: Controllable High-Resolution Portrait Video Style Transfer

Shuai Yang, Liming Jiang, Ziwei Liu, Chen Change Loy
{"title":"VToonify: Controllable High-Resolution Portrait Video Style Transfer","authors":"Shuai Yang, Liming Jiang, Ziwei Liu, Chen Change Loy","doi":"10.48550/arXiv.2209.11224","DOIUrl":null,"url":null,"abstract":"Generating high-quality artistic portrait videos is an important and desirable task in computer graphics and vision. Although a series of successful portrait image toonification models built upon the powerful StyleGAN have been proposed, these image-oriented methods have obvious limitations when applied to videos, such as the fixed frame size, the requirement of face alignment, missing non-facial details and temporal inconsistency. In this work, we investigate the challenging controllable high-resolution portrait video style transfer by introducing a novel VToonify framework. Specifically, VToonify leverages the mid- and high-resolution layers of StyleGAN to render high-quality artistic portraits based on the multi-scale content features extracted by an encoder to better preserve the frame details. The resulting fully convolutional architecture accepts non-aligned faces in videos of variable size as input, contributing to complete face regions with natural motions in the output. Our framework is compatible with existing StyleGAN-based image toonification models to extend them to video toonification, and inherits appealing features of these models for flexible style control on color and intensity. This work presents two instantiations of VToonify built upon Toonify and DualStyleGAN for collection-based and exemplar-based portrait video style transfer, respectively. Extensive experimental results demonstrate the effectiveness of our proposed VToonify framework over existing methods in generating high-quality and temporally-coherent artistic portrait videos with flexible style controls.","PeriodicalId":7121,"journal":{"name":"ACM Trans. Graph.","volume":"03 1","pages":"203:1-203:15"},"PeriodicalIF":0.0000,"publicationDate":"2022-09-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"20","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"ACM Trans. Graph.","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.48550/arXiv.2209.11224","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 20

Abstract

Generating high-quality artistic portrait videos is an important and desirable task in computer graphics and vision. Although a series of successful portrait image toonification models built upon the powerful StyleGAN have been proposed, these image-oriented methods have obvious limitations when applied to videos, such as the fixed frame size, the requirement of face alignment, missing non-facial details and temporal inconsistency. In this work, we investigate the challenging controllable high-resolution portrait video style transfer by introducing a novel VToonify framework. Specifically, VToonify leverages the mid- and high-resolution layers of StyleGAN to render high-quality artistic portraits based on the multi-scale content features extracted by an encoder to better preserve the frame details. The resulting fully convolutional architecture accepts non-aligned faces in videos of variable size as input, contributing to complete face regions with natural motions in the output. Our framework is compatible with existing StyleGAN-based image toonification models to extend them to video toonification, and inherits appealing features of these models for flexible style control on color and intensity. This work presents two instantiations of VToonify built upon Toonify and DualStyleGAN for collection-based and exemplar-based portrait video style transfer, respectively. Extensive experimental results demonstrate the effectiveness of our proposed VToonify framework over existing methods in generating high-quality and temporally-coherent artistic portrait videos with flexible style controls.
查看原文
分享 分享
微信好友 朋友圈 QQ好友 复制链接
本刊更多论文
VToonify:可控的高分辨率肖像视频风格转移
在计算机图形学和视觉中,生成高质量的艺术肖像视频是一项重要而理想的任务。虽然已经提出了一系列成功的基于强大的StyleGAN的人像图像化模型,但这些面向图像的方法在应用于视频时存在明显的局限性,如固定的帧大小、面部对齐要求、缺少非面部细节和时间不一致。在这项工作中,我们通过引入一种新的VToonify框架来研究具有挑战性的可控高分辨率人像视频风格转移。具体来说,VToonify利用StyleGAN的中高分辨率层,基于编码器提取的多尺度内容特征来渲染高质量的艺术肖像,以更好地保留帧细节。由此产生的全卷积架构接受可变大小视频中的未对齐面部作为输入,从而在输出中贡献具有自然运动的完整面部区域。我们的框架与现有的基于stylegan的图像色调化模型兼容,将其扩展到视频色调化,并继承了这些模型在颜色和强度上灵活的风格控制的吸引人的特性。这项工作提出了VToonify的两个实例,分别建立在Toonify和DualStyleGAN的基础上,用于基于集合和基于示例的人像视频风格转移。大量的实验结果表明,我们提出的VToonify框架在生成具有灵活样式控制的高质量和时间连贯的艺术肖像视频方面优于现有方法的有效性。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 去求助
来源期刊
自引率
0.00%
发文量
0
期刊最新文献
LuisaRender: A High-Performance Rendering Framework with Layered and Unified Interfaces on Stream Architectures BoolSurf: Boolean Operations on Surfaces SkinMixer: Blending 3D Animated Models PopStage: The Generation of Stage Cross-Editing Video Based on Spatio-Temporal Matching QuadStream: A Quad-Based Scene Streaming Architecture for Novel Viewpoint Reconstruction
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
现在去查看 取消
×
提示
确定
0
微信
客服QQ
Book学术公众号 扫码关注我们
反馈
×
意见反馈
请填写您的意见或建议
请填写您的手机或邮箱
已复制链接
已复制链接
快去分享给好友吧!
我知道了
×
扫码分享
扫码分享
Book学术官方微信
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术
文献互助 智能选刊 最新文献 互助须知 联系我们:info@booksci.cn
Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。
Copyright © 2023 Book学术 All rights reserved.
ghs 京公网安备 11010802042870号 京ICP备2023020795号-1