Hangyan Zhu, Shaohui Liu, Ming Liu, Zifei Yan, Wangmeng Zuo
{"title":"De2Net:利用特征解卷积和内核分解修复显示不足的摄像头图像","authors":"Hangyan Zhu, Shaohui Liu, Ming Liu, Zifei Yan, Wangmeng Zuo","doi":"10.1016/j.cviu.2024.104028","DOIUrl":null,"url":null,"abstract":"<div><p>While the under-display camera (UDC) system provides an effective solution for notch-free full-screen displays, it inevitably causes severe image quality degradation due to the diffraction phenomenon. Recent methods have achieved decent performance with deep neural networks, yet the characteristic of the point spread function (PSF) is less studied. In this paper, considering the large support and spatial inconsistency of PSF, we propose De<span><math><msup><mrow></mrow><mrow><mn>2</mn></mrow></msup></math></span>Net for UDC image restoration with feature <em>de</em>convolution and kernel <em>de</em>composition. In terms of feature deconvolution, we introduce Wiener deconvolution as a preliminary process, which alleviates feature entanglement caused by the large PSF support. Besides, the deconvolution kernel can be learned from training images, eliminating the tedious PSF-obtaining process. As for kernel decomposition, we observe regular patterns for PSFs at different positions. Thus, with a kernel prediction network (KPN) deployed for handling the spatial inconsistency problem, we improve it from two aspects, <em>i.e.</em>, (i) decomposing the predicted kernels into a set of bases and weights, (ii) decomposing kernels into groups with different dilation rates. These modifications largely improve the receptive field under certain memory limits. Extensive experiments on three commonly used UDC datasets show that De<span><math><msup><mrow></mrow><mrow><mn>2</mn></mrow></msup></math></span>Net outperforms existing methods both quantitatively and qualitatively. Source code and pre-trained models are available at <span>https://github.com/HyZhu39/De2Net</span><svg><path></path></svg>.</p></div>","PeriodicalId":50633,"journal":{"name":"Computer Vision and Image Understanding","volume":null,"pages":null},"PeriodicalIF":4.3000,"publicationDate":"2024-04-25","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":"{\"title\":\"De2Net: Under-display camera image restoration with feature deconvolution and kernel decomposition\",\"authors\":\"Hangyan Zhu, Shaohui Liu, Ming Liu, Zifei Yan, Wangmeng Zuo\",\"doi\":\"10.1016/j.cviu.2024.104028\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"<div><p>While the under-display camera (UDC) system provides an effective solution for notch-free full-screen displays, it inevitably causes severe image quality degradation due to the diffraction phenomenon. Recent methods have achieved decent performance with deep neural networks, yet the characteristic of the point spread function (PSF) is less studied. In this paper, considering the large support and spatial inconsistency of PSF, we propose De<span><math><msup><mrow></mrow><mrow><mn>2</mn></mrow></msup></math></span>Net for UDC image restoration with feature <em>de</em>convolution and kernel <em>de</em>composition. In terms of feature deconvolution, we introduce Wiener deconvolution as a preliminary process, which alleviates feature entanglement caused by the large PSF support. Besides, the deconvolution kernel can be learned from training images, eliminating the tedious PSF-obtaining process. As for kernel decomposition, we observe regular patterns for PSFs at different positions. Thus, with a kernel prediction network (KPN) deployed for handling the spatial inconsistency problem, we improve it from two aspects, <em>i.e.</em>, (i) decomposing the predicted kernels into a set of bases and weights, (ii) decomposing kernels into groups with different dilation rates. These modifications largely improve the receptive field under certain memory limits. Extensive experiments on three commonly used UDC datasets show that De<span><math><msup><mrow></mrow><mrow><mn>2</mn></mrow></msup></math></span>Net outperforms existing methods both quantitatively and qualitatively. Source code and pre-trained models are available at <span>https://github.com/HyZhu39/De2Net</span><svg><path></path></svg>.</p></div>\",\"PeriodicalId\":50633,\"journal\":{\"name\":\"Computer Vision and Image Understanding\",\"volume\":null,\"pages\":null},\"PeriodicalIF\":4.3000,\"publicationDate\":\"2024-04-25\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"Computer Vision and Image Understanding\",\"FirstCategoryId\":\"94\",\"ListUrlMain\":\"https://www.sciencedirect.com/science/article/pii/S1077314224001097\",\"RegionNum\":3,\"RegionCategory\":\"计算机科学\",\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"Q2\",\"JCRName\":\"COMPUTER SCIENCE, ARTIFICIAL INTELLIGENCE\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"Computer Vision and Image Understanding","FirstCategoryId":"94","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S1077314224001097","RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q2","JCRName":"COMPUTER SCIENCE, ARTIFICIAL INTELLIGENCE","Score":null,"Total":0}
De2Net: Under-display camera image restoration with feature deconvolution and kernel decomposition
While the under-display camera (UDC) system provides an effective solution for notch-free full-screen displays, it inevitably causes severe image quality degradation due to the diffraction phenomenon. Recent methods have achieved decent performance with deep neural networks, yet the characteristic of the point spread function (PSF) is less studied. In this paper, considering the large support and spatial inconsistency of PSF, we propose DeNet for UDC image restoration with feature deconvolution and kernel decomposition. In terms of feature deconvolution, we introduce Wiener deconvolution as a preliminary process, which alleviates feature entanglement caused by the large PSF support. Besides, the deconvolution kernel can be learned from training images, eliminating the tedious PSF-obtaining process. As for kernel decomposition, we observe regular patterns for PSFs at different positions. Thus, with a kernel prediction network (KPN) deployed for handling the spatial inconsistency problem, we improve it from two aspects, i.e., (i) decomposing the predicted kernels into a set of bases and weights, (ii) decomposing kernels into groups with different dilation rates. These modifications largely improve the receptive field under certain memory limits. Extensive experiments on three commonly used UDC datasets show that DeNet outperforms existing methods both quantitatively and qualitatively. Source code and pre-trained models are available at https://github.com/HyZhu39/De2Net.
期刊介绍:
The central focus of this journal is the computer analysis of pictorial information. Computer Vision and Image Understanding publishes papers covering all aspects of image analysis from the low-level, iconic processes of early vision to the high-level, symbolic processes of recognition and interpretation. A wide range of topics in the image understanding area is covered, including papers offering insights that differ from predominant views.
Research Areas Include:
• Theory
• Early vision
• Data structures and representations
• Shape
• Range
• Motion
• Matching and recognition
• Architecture and languages
• Vision systems