Pub Date : 2024-06-07DOI: 10.1109/TBC.2024.3391025
Simin Keykhosravi;Ebrahim Bedeer
This paper investigates doubly-selective (i.e., time- and frequency-selective) channel estimation in faster-than-Nyquist (FTN) signaling HF communications. In particular, we propose a novel IM-based channel estimation algorithm for FTN signaling HF communications including pilot sequence placement (PSP) and pilot sequence location identification (PSLI) algorithms. At the transmitter, we propose the PSP algorithm that utilizes the locations of pilot sequences to carry additional information bits, thereby improving the SE of HF communications. HF channels have two non-zero independent fading paths with specific fixed delay spread and frequency spread characteristics as outlined in the Union Radio communication Sector (ITU-R) F.1487 and F.520. Having said that, based on the aforementioned properties of the HF channels and the favorable auto-correlation characteristics of the optimal pilot sequence, we propose a novel PSLI algorithm that effectively identifies the pilot sequence location within a given frame at the receiver. This is achieved by showing that the square of the absolute value of the cross-correlation between the received symbols and the pilot sequence consists of a scaled version of the square of the absolute value of the auto-correlation of the pilot sequence weighted by the gain of the corresponding HF channel path. Simulation results show very low pilot sequence location identification errors for HF channels. Our simulation results show a 6 dB improvement in the MSE of the channel estimation as well as about 3.5 dB BER improvement of FTN signaling along with an enhancement in SE compared to the method in Ishihara and Sugiura (2017). We also achieved an enhancement in SE compared to the work in Keykhosravi and Bedeer (2023) while maintaining comparable MSE of the channel estimation and BER performance.
{"title":"IM-Based Pilot-Assisted Channel Estimation for FTN Signaling HF Communications","authors":"Simin Keykhosravi;Ebrahim Bedeer","doi":"10.1109/TBC.2024.3391025","DOIUrl":"10.1109/TBC.2024.3391025","url":null,"abstract":"This paper investigates doubly-selective (i.e., time- and frequency-selective) channel estimation in faster-than-Nyquist (FTN) signaling HF communications. In particular, we propose a novel IM-based channel estimation algorithm for FTN signaling HF communications including pilot sequence placement (PSP) and pilot sequence location identification (PSLI) algorithms. At the transmitter, we propose the PSP algorithm that utilizes the locations of pilot sequences to carry additional information bits, thereby improving the SE of HF communications. HF channels have two non-zero independent fading paths with specific fixed delay spread and frequency spread characteristics as outlined in the Union Radio communication Sector (ITU-R) F.1487 and F.520. Having said that, based on the aforementioned properties of the HF channels and the favorable auto-correlation characteristics of the optimal pilot sequence, we propose a novel PSLI algorithm that effectively identifies the pilot sequence location within a given frame at the receiver. This is achieved by showing that the square of the absolute value of the cross-correlation between the received symbols and the pilot sequence consists of a scaled version of the square of the absolute value of the auto-correlation of the pilot sequence weighted by the gain of the corresponding HF channel path. Simulation results show very low pilot sequence location identification errors for HF channels. Our simulation results show a 6 dB improvement in the MSE of the channel estimation as well as about 3.5 dB BER improvement of FTN signaling along with an enhancement in SE compared to the method in Ishihara and Sugiura (2017). We also achieved an enhancement in SE compared to the work in Keykhosravi and Bedeer (2023) while maintaining comparable MSE of the channel estimation and BER performance.","PeriodicalId":13159,"journal":{"name":"IEEE Transactions on Broadcasting","volume":"70 3","pages":"774-791"},"PeriodicalIF":3.2,"publicationDate":"2024-06-07","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141940845","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-06-03DOI: 10.1109/TBC.2024.3402380
George Henrique Maranhão Garcia de Oliveira;Gustavo de Melo Valeira;Cristiano Akamine
In Brazil, the Television (TV) 3.0 project has been underway since 2020 and is currently in its third phase. The aim of this project is to study, test and validate state-of-the-art technologies in order to define the techniques that will make up the next-generation of Brazilian Digital Terrestrial Television Broadcasting (DTTB) System. All the technologies involved in this system must be compatible with the transportation method defined in Phase 02 of the project: the Real-Time Object Delivery over Unidirectional Transport (ROUTE)/Dynamic Adaptive Streaming over HTTP (DASH) method from the Advanced Television Systems Committee (ATSC) 3.0 standard. Therefore, this paper proposes the use of the ROUTE/DASH transportation method in the Advanced Integrated Services Digital Broadcasting Terrestrial (ISDB-T) system, presenting the theory involved and the results obtained in the first transmission carried out involving the two aforementioned technologies.
{"title":"A Proposal to Use ROUTE/DASH in the Advanced ISDB-T","authors":"George Henrique Maranhão Garcia de Oliveira;Gustavo de Melo Valeira;Cristiano Akamine","doi":"10.1109/TBC.2024.3402380","DOIUrl":"10.1109/TBC.2024.3402380","url":null,"abstract":"In Brazil, the Television (TV) 3.0 project has been underway since 2020 and is currently in its third phase. The aim of this project is to study, test and validate state-of-the-art technologies in order to define the techniques that will make up the next-generation of Brazilian Digital Terrestrial Television Broadcasting (DTTB) System. All the technologies involved in this system must be compatible with the transportation method defined in Phase 02 of the project: the Real-Time Object Delivery over Unidirectional Transport (ROUTE)/Dynamic Adaptive Streaming over HTTP (DASH) method from the Advanced Television Systems Committee (ATSC) 3.0 standard. Therefore, this paper proposes the use of the ROUTE/DASH transportation method in the Advanced Integrated Services Digital Broadcasting Terrestrial (ISDB-T) system, presenting the theory involved and the results obtained in the first transmission carried out involving the two aforementioned technologies.","PeriodicalId":13159,"journal":{"name":"IEEE Transactions on Broadcasting","volume":"70 3","pages":"935-944"},"PeriodicalIF":3.2,"publicationDate":"2024-06-03","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141940843","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
The latest technological developments have fueled revolutionary changes and improvements in wireless communication systems. Among them, mmWave spectrum exploitation stands out for its ability to deliver ultra-high data rates. However, its full adoption beyond fifth generation multicast systems (5G+/6G) remains hampered, mainly due to mobility robustness issues. In this work, we propose a solution to address the problem of efficient sidelink-assisted multicasting in mobile multimode systems, specifically by considering the possibility of jointly utilizing sidelink/device-to-device (D2D), unicast, and multicast transmissions to improve service delivery. To overcome the complexity problem in finding the optimal solution for user-mode binding, we introduce a pre-optimization step called multicast group formation (MGF). Through a clustering technique based on unsupervised machine learning, MGF allows to reduce the complexity of solving the sidelink-assisted multiple modes mmWave (SA3M) problem. A detailed analysis of the impact of various system parameters on performance is conducted, and numerical evidence of the complexity/performance trade-off and its dependence on mobility patterns and user distribution is provided. Particularly, our proposed solution achieves a network throughput improvement of up to 32% over state-of-the-art schemes while ensuring the lowest computational time. Finally, the results demonstrate that an effective balance between power consumption and latency can be achieved through appropriate adjustments of transmit power and bandwidth.
{"title":"Beyond Complexity Limits: Machine Learning for Sidelink-Assisted mmWave Multicasting in 6G","authors":"Nadezhda Chukhno;Olga Chukhno;Sara Pizzi;Antonella Molinaro;Antonio Iera;Giuseppe Araniti","doi":"10.1109/TBC.2024.3382959","DOIUrl":"10.1109/TBC.2024.3382959","url":null,"abstract":"The latest technological developments have fueled revolutionary changes and improvements in wireless communication systems. Among them, mmWave spectrum exploitation stands out for its ability to deliver ultra-high data rates. However, its full adoption beyond fifth generation multicast systems (5G+/6G) remains hampered, mainly due to mobility robustness issues. In this work, we propose a solution to address the problem of efficient sidelink-assisted multicasting in mobile multimode systems, specifically by considering the possibility of jointly utilizing sidelink/device-to-device (D2D), unicast, and multicast transmissions to improve service delivery. To overcome the complexity problem in finding the optimal solution for user-mode binding, we introduce a pre-optimization step called multicast group formation (MGF). Through a clustering technique based on unsupervised machine learning, MGF allows to reduce the complexity of solving the sidelink-assisted multiple modes mmWave (SA3M) problem. A detailed analysis of the impact of various system parameters on performance is conducted, and numerical evidence of the complexity/performance trade-off and its dependence on mobility patterns and user distribution is provided. Particularly, our proposed solution achieves a network throughput improvement of up to 32% over state-of-the-art schemes while ensuring the lowest computational time. Finally, the results demonstrate that an effective balance between power consumption and latency can be achieved through appropriate adjustments of transmit power and bandwidth.","PeriodicalId":13159,"journal":{"name":"IEEE Transactions on Broadcasting","volume":"70 3","pages":"1076-1090"},"PeriodicalIF":3.2,"publicationDate":"2024-04-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10513425","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140839613","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-04-22DOI: 10.1109/TBC.2024.3380474
Sungjun Ahn;Hyun-Jeong Yim;Youngwan Lee;Sung-Ik Park
This paper introduces a media service model that exploits artificial intelligence (AI) video generators at the receive end. This proposal deviates from the traditional multimedia ecosystem, completely relying on in-house production, by shifting part of the content creation onto the receiver. We bring a semantic process into the framework, allowing the distribution network to provide service elements that prompt the content generator rather than distributing encoded data of fully finished programs. The service elements include fine-tailored text descriptions, lightweight image data of some objects, or application programming interfaces, comprehensively referred to as semantic sources, and the user terminal translates the received semantic data into video frames. Empowered by the random nature of generative AI, users can experience super-personalized services accordingly. The proposed idea incorporates situations in which the user receives different service providers’ element packages, either in a sequence over time or multiple packages at the same time. Given promised in-context coherence and content integrity, the combinatory dynamics will amplify the service diversity, allowing the users to always chance upon new experiences. This work particularly aims at short-form videos and advertisements, which the users would easily feel fatigued by seeing the same frame sequence every time. In those use cases, the content provider’s role will be recast as scripting semantic sources, transformed from a thorough producer. Overall, this work explores a new form of media ecosystem facilitated by receiver-embedded generative models, featuring both random content dynamics and enhanced delivery efficiency simultaneously.
{"title":"Dynamic and Super-Personalized Media Ecosystem Driven by Generative AI: Unpredictable Plays Never Repeating the Same","authors":"Sungjun Ahn;Hyun-Jeong Yim;Youngwan Lee;Sung-Ik Park","doi":"10.1109/TBC.2024.3380474","DOIUrl":"10.1109/TBC.2024.3380474","url":null,"abstract":"This paper introduces a media service model that exploits artificial intelligence (AI) video generators at the receive end. This proposal deviates from the traditional multimedia ecosystem, completely relying on in-house production, by shifting part of the content creation onto the receiver. We bring a semantic process into the framework, allowing the distribution network to provide service elements that prompt the content generator rather than distributing encoded data of fully finished programs. The service elements include fine-tailored text descriptions, lightweight image data of some objects, or application programming interfaces, comprehensively referred to as semantic sources, and the user terminal translates the received semantic data into video frames. Empowered by the random nature of generative AI, users can experience super-personalized services accordingly. The proposed idea incorporates situations in which the user receives different service providers’ element packages, either in a sequence over time or multiple packages at the same time. Given promised in-context coherence and content integrity, the combinatory dynamics will amplify the service diversity, allowing the users to always chance upon new experiences. This work particularly aims at short-form videos and advertisements, which the users would easily feel fatigued by seeing the same frame sequence every time. In those use cases, the content provider’s role will be recast as scripting semantic sources, transformed from a thorough producer. Overall, this work explores a new form of media ecosystem facilitated by receiver-embedded generative models, featuring both random content dynamics and enhanced delivery efficiency simultaneously.","PeriodicalId":13159,"journal":{"name":"IEEE Transactions on Broadcasting","volume":"70 3","pages":"980-994"},"PeriodicalIF":3.2,"publicationDate":"2024-04-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140637223","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Most existing learning-based methods for stereoscopic image super-resolution rely on a great number of high-resolution stereoscopic images as labels. To alleviate the problem of data dependency, this paper proposes a self-supervised pretraining-based method for stereoscopic image super-resolution (SelfSSR). Specifically, to develop a self-supervised pretext task for stereoscopic images, a parallax-aware masking strategy (PAMS) is designed to adaptively mask matching areas of the left and right views. With PAMS, the network is encouraged to effectively predict missing information of input images. Besides, a cross-view Transformer module (CVTM) is presented to aggregate the intra-view and inter-view information simultaneously for stereoscopic image reconstruction. Meanwhile, the cross-attention map learned by CVTM is utilized to guide the masking process in PAMS. Comparative results on four datasets show that the proposed SelfSSR achieves state-of-the-art performance by using only 10% of labeled training data.
{"title":"Self-Supervised Pretraining for Stereoscopic Image Super-Resolution With Parallax-Aware Masking","authors":"Zhe Zhang;Jianjun Lei;Bo Peng;Jie Zhu;Qingming Huang","doi":"10.1109/TBC.2024.3382960","DOIUrl":"10.1109/TBC.2024.3382960","url":null,"abstract":"Most existing learning-based methods for stereoscopic image super-resolution rely on a great number of high-resolution stereoscopic images as labels. To alleviate the problem of data dependency, this paper proposes a self-supervised pretraining-based method for stereoscopic image super-resolution (SelfSSR). Specifically, to develop a self-supervised pretext task for stereoscopic images, a parallax-aware masking strategy (PAMS) is designed to adaptively mask matching areas of the left and right views. With PAMS, the network is encouraged to effectively predict missing information of input images. Besides, a cross-view Transformer module (CVTM) is presented to aggregate the intra-view and inter-view information simultaneously for stereoscopic image reconstruction. Meanwhile, the cross-attention map learned by CVTM is utilized to guide the masking process in PAMS. Comparative results on four datasets show that the proposed SelfSSR achieves state-of-the-art performance by using only 10% of labeled training data.","PeriodicalId":13159,"journal":{"name":"IEEE Transactions on Broadcasting","volume":"70 2","pages":"482-491"},"PeriodicalIF":4.5,"publicationDate":"2024-04-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140637362","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
The widespread use of live streaming necessitates low-latency requirements for the processing and transmission of virtual reality (VR) videos. This paper introduces a prototype system for low-latency VR video processing and transmission that exploits edge computing to harness the computational power of edge servers. This approach enables efficient video preprocessing and facilitates closer-to-user multicast video distribution. Despite edge computing’s potential, managing large-scale access, addressing differentiated channel conditions, and accommodating diverse user viewports pose significant challenges for VR video transcoding and scheduling. To tackle these challenges, our system utilizes dual-edge servers for video transcoding and slicing, thereby markedly improving the viewing experience compared to traditional cloud-based systems. Additionally, we devise a low-complexity greedy algorithm for multi-edge and multi-user VR video offloading distribution, employing the results of bitrate decisions to guide video transcoding inversely. Simulation results reveal that our strategy significantly enhances system utility by 44.77% over existing state-of-the-art schemes that do not utilize edge servers while reducing processing time by 58.54%.
{"title":"Low-Latency VR Video Processing-Transmitting System Based on Edge Computing","authors":"Nianzhen Gao;Jiaxi Zhou;Guoan Wan;Xinhai Hua;Ting Bi;Tao Jiang","doi":"10.1109/TBC.2024.3380455","DOIUrl":"10.1109/TBC.2024.3380455","url":null,"abstract":"The widespread use of live streaming necessitates low-latency requirements for the processing and transmission of virtual reality (VR) videos. This paper introduces a prototype system for low-latency VR video processing and transmission that exploits edge computing to harness the computational power of edge servers. This approach enables efficient video preprocessing and facilitates closer-to-user multicast video distribution. Despite edge computing’s potential, managing large-scale access, addressing differentiated channel conditions, and accommodating diverse user viewports pose significant challenges for VR video transcoding and scheduling. To tackle these challenges, our system utilizes dual-edge servers for video transcoding and slicing, thereby markedly improving the viewing experience compared to traditional cloud-based systems. Additionally, we devise a low-complexity greedy algorithm for multi-edge and multi-user VR video offloading distribution, employing the results of bitrate decisions to guide video transcoding inversely. Simulation results reveal that our strategy significantly enhances system utility by 44.77% over existing state-of-the-art schemes that do not utilize edge servers while reducing processing time by 58.54%.","PeriodicalId":13159,"journal":{"name":"IEEE Transactions on Broadcasting","volume":"70 3","pages":"862-871"},"PeriodicalIF":3.2,"publicationDate":"2024-04-11","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140578571","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-04-11DOI: 10.1109/TBC.2024.3382949
Fei Zhou;Wei Sheng;Zitao Lu;Guoping Qiu
Video super-resolution (SR) has important real world applications such as enhancing viewing experiences of legacy low-resolution videos on high resolution display devices. However, there are no visual quality assessment (VQA) models specifically designed for evaluating SR videos while such models are crucially important both for advancing video SR algorithms and for viewing quality assurance. This paper addresses this gap. We start by contributing the first video super-resolution quality assessment database (VSR-QAD) which contains 2,260 SR videos annotated with mean opinion score (MOS) labels collected through an approximately 400 man-hours psychovisual experiment by a total of 190 subjects. We then build on the new VSR-QAD and develop the first VQA model specifically designed for evaluating SR videos. The model features a two-stream convolutional neural network architecture and a two-stage training algorithm designed for extracting spatial and temporal features characterizing the quality of SR videos. We present experimental results and data analysis to demonstrate the high data quality of VSR-QAD and the effectiveness of the new VQA model for measuring the visual quality of SR videos. The new database and the code of the proposed model will be available online at https://github.com/key1cdc/VSRQAD .
视频超分辨率（SR）在现实世界中有着重要的应用，例如在高分辨率显示设备上增强传统低分辨率视频的观看体验。然而，目前还没有专门用于评估 SR 视频的视觉质量评估 (VQA) 模型，而这类模型对于推进视频 SR 算法和保证观看质量都至关重要。本文正是为了弥补这一空白。首先，我们提供了第一个视频超分辨率质量评估数据库（VSR-QAD），该数据库包含 2,260 个 SR 视频，这些视频标注了平均意见分（MOS）标签，这些标签是由 190 名受试者通过约 400 个工时的心理视觉实验收集的。然后，我们以新的 VSR-QAD 为基础，开发了首个专门用于评估 SR 视频的 VQA 模型。该模型采用双流卷积神经网络架构和两阶段训练算法，旨在提取表征 SR 视频质量的空间和时间特征。我们展示了实验结果和数据分析，以证明 VSR-QAD 的高数据质量和新 VQA 模型在测量 SR 视频视觉质量方面的有效性。新数据库和拟议模型的代码将在 https://github.com/key1cdc/VSRQAD 上在线提供。
{"title":"A Database and Model for the Visual Quality Assessment of Super-Resolution Videos","authors":"Fei Zhou;Wei Sheng;Zitao Lu;Guoping Qiu","doi":"10.1109/TBC.2024.3382949","DOIUrl":"10.1109/TBC.2024.3382949","url":null,"abstract":"Video super-resolution (SR) has important real world applications such as enhancing viewing experiences of legacy low-resolution videos on high resolution display devices. However, there are no visual quality assessment (VQA) models specifically designed for evaluating SR videos while such models are crucially important both for advancing video SR algorithms and for viewing quality assurance. This paper addresses this gap. We start by contributing the first video super-resolution quality assessment database (VSR-QAD) which contains 2,260 SR videos annotated with mean opinion score (MOS) labels collected through an approximately 400 man-hours psychovisual experiment by a total of 190 subjects. We then build on the new VSR-QAD and develop the first VQA model specifically designed for evaluating SR videos. The model features a two-stream convolutional neural network architecture and a two-stage training algorithm designed for extracting spatial and temporal features characterizing the quality of SR videos. We present experimental results and data analysis to demonstrate the high data quality of VSR-QAD and the effectiveness of the new VQA model for measuring the visual quality of SR videos. The new database and the code of the proposed model will be available online at \u0000<uri>https://github.com/key1cdc/VSRQAD</uri>\u0000.","PeriodicalId":13159,"journal":{"name":"IEEE Transactions on Broadcasting","volume":"70 2","pages":"516-532"},"PeriodicalIF":4.5,"publicationDate":"2024-04-11","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140578579","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-04-10DOI: 10.1109/TBC.2024.3380435
Zizhuang Zou;Mao Ye;Xue Li;Luping Ji;Ce Zhu
With the popularity of panoramic cameras and head mount displays, many 360° videos have been recorded. Due to the geometric distortion and boundary discontinuity of 2D projection of 360° video, traditional 2D lossy video compression technology always generates more artifacts. Therefore, it is necessary to enhance the quality of compressed 360° video. However, 360° video characteristics make traditional 2D enhancement models cannot work properly. So the previous work tries to obtain the viewport sequence with smaller geometric distortions for enhancement. But such sequence is difficult to be obtained and the trained enhancement model cannot be well adapted to a new dataset. To address these issues, we propose a Stable viewport-based Unsupervised compressed 360° video Quality Enhancement (SUQE) method. Our method consists of two stages. In the first stage, a new data preparation module is proposed which adopts saliency-based data augmentation and viewport cropping techniques to generate training dataset. A standard 2D enhancement model is trained based on this dataset. For transferring the trained enhancement model to the target dataset, a shift prediction module is designed, which will crop a shifted viewport clip as supervision signal for model adaptation. For the second stage, by comparing the differences between the current enhanced original and shifted frames, the Mean Teacher framework is employed to further fine-tune the enhancement model. Experiment results confirm that our method achieves satisfactory performance on the public dataset. The relevant models and code will be released.
{"title":"Stable Viewport-Based Unsupervised Compressed 360° Video Quality Enhancement","authors":"Zizhuang Zou;Mao Ye;Xue Li;Luping Ji;Ce Zhu","doi":"10.1109/TBC.2024.3380435","DOIUrl":"10.1109/TBC.2024.3380435","url":null,"abstract":"With the popularity of panoramic cameras and head mount displays, many 360° videos have been recorded. Due to the geometric distortion and boundary discontinuity of 2D projection of 360° video, traditional 2D lossy video compression technology always generates more artifacts. Therefore, it is necessary to enhance the quality of compressed 360° video. However, 360° video characteristics make traditional 2D enhancement models cannot work properly. So the previous work tries to obtain the viewport sequence with smaller geometric distortions for enhancement. But such sequence is difficult to be obtained and the trained enhancement model cannot be well adapted to a new dataset. To address these issues, we propose a Stable viewport-based Unsupervised compressed 360° video Quality Enhancement (SUQE) method. Our method consists of two stages. In the first stage, a new data preparation module is proposed which adopts saliency-based data augmentation and viewport cropping techniques to generate training dataset. A standard 2D enhancement model is trained based on this dataset. For transferring the trained enhancement model to the target dataset, a shift prediction module is designed, which will crop a shifted viewport clip as supervision signal for model adaptation. For the second stage, by comparing the differences between the current enhanced original and shifted frames, the Mean Teacher framework is employed to further fine-tune the enhancement model. Experiment results confirm that our method achieves satisfactory performance on the public dataset. The relevant models and code will be released.","PeriodicalId":13159,"journal":{"name":"IEEE Transactions on Broadcasting","volume":"70 2","pages":"607-619"},"PeriodicalIF":4.5,"publicationDate":"2024-04-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140578576","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-04-01DOI: 10.1109/TBC.2024.3374103
Ge Li;Jianjun Lei;Zhaoqing Pan;Bo Peng;Nam Ling
Due to the fact that depth video contains large similar smooth content, the depth video frame could be selectively generated at the decoder side without being encoded and transmitted at the encoder side, so as to achieve a significant improvement in coding efficiency. This paper proposes a deep frame generation-based depth video inter coding method to efficiently compress the depth video. To reduce temporal redundancies of the depth video, the proposed method encodes depth key frames and directly generates the reconstruction of depth non-key frames. Moreover, a warping-based frame generation network with boundary awareness (Ba-WFGNet) is designed to generate high-quality depth non-key frames at the decoder side. In the Ba-WFGNet, the temporal correlations among depth frames are utilized to generate the coarse depth non-key frame in a warping manner. Then, considering the boundary quality of depth video has an important impact on view synthesis, a boundary-aware refinement module is designed to further refine the coarse depth non-key frame for high-quality boundaries. The proposed method is implemented into MIV, and experimental results verify that the proposed method achieves superior coding efficiency.
{"title":"Depth Video Inter Coding Based on Deep Frame Generation","authors":"Ge Li;Jianjun Lei;Zhaoqing Pan;Bo Peng;Nam Ling","doi":"10.1109/TBC.2024.3374103","DOIUrl":"10.1109/TBC.2024.3374103","url":null,"abstract":"Due to the fact that depth video contains large similar smooth content, the depth video frame could be selectively generated at the decoder side without being encoded and transmitted at the encoder side, so as to achieve a significant improvement in coding efficiency. This paper proposes a deep frame generation-based depth video inter coding method to efficiently compress the depth video. To reduce temporal redundancies of the depth video, the proposed method encodes depth key frames and directly generates the reconstruction of depth non-key frames. Moreover, a warping-based frame generation network with boundary awareness (Ba-WFGNet) is designed to generate high-quality depth non-key frames at the decoder side. In the Ba-WFGNet, the temporal correlations among depth frames are utilized to generate the coarse depth non-key frame in a warping manner. Then, considering the boundary quality of depth video has an important impact on view synthesis, a boundary-aware refinement module is designed to further refine the coarse depth non-key frame for high-quality boundaries. The proposed method is implemented into MIV, and experimental results verify that the proposed method achieves superior coding efficiency.","PeriodicalId":13159,"journal":{"name":"IEEE Transactions on Broadcasting","volume":"70 2","pages":"708-718"},"PeriodicalIF":4.5,"publicationDate":"2024-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140578578","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Facial retouching, aiming at enhancing an individual’s appearance digitally, has become popular in many parts of human life, such as personal entertainment, commercial advertising, etc. However, excessive use of facial retouching can affect public aesthetic values and accordingly induce issues of mental health. There is a growing need for comprehensive quality assessment of Retouched Face (RF) images. This paper aims to advance this topic from both subjective and objective studies. Firstly, we generate 2,500 RF images by retouching 250 high-quality face images from multiple attributes (i.e., eyes, nose, mouth, and facial shape) with different photo-editing tools. After that, we carry out a series of subjective experiments to evaluate the quality of multi-attribute RF images from various perspectives, and construct the Multi-Attribute Retouched Face Database (MARFD) with multi-labels. Secondly, considering that retouching alters the facial morphology, we introduce a multi-task learning based No-Reference (NR) Image Quality Assessment (IQA) method, named MTNet. Specifically, to capture high-level semantic information associated with geometric changes, MTNet treats the alteration degree estimation of retouching attributes as auxiliary tasks for the main task (i.e., the overall quality prediction). In addition, inspired by the perceptual effects of viewing distance, MTNet utilizes a multi-scale data augmentation strategy during network training to help the network better understand the distortions. Experimental results on MARFD show that our MTNet correlates well with subjective ratings and outperforms 16 state-of-the-art NR-IQA methods.
{"title":"Subjective and Objective Quality Assessment of Multi-Attribute Retouched Face Images","authors":"Guanghui Yue;Honglv Wu;Weiqing Yan;Tianwei Zhou;Hantao Liu;Wei Zhou","doi":"10.1109/TBC.2024.3374043","DOIUrl":"10.1109/TBC.2024.3374043","url":null,"abstract":"Facial retouching, aiming at enhancing an individual’s appearance digitally, has become popular in many parts of human life, such as personal entertainment, commercial advertising, etc. However, excessive use of facial retouching can affect public aesthetic values and accordingly induce issues of mental health. There is a growing need for comprehensive quality assessment of Retouched Face (RF) images. This paper aims to advance this topic from both subjective and objective studies. Firstly, we generate 2,500 RF images by retouching 250 high-quality face images from multiple attributes (i.e., eyes, nose, mouth, and facial shape) with different photo-editing tools. After that, we carry out a series of subjective experiments to evaluate the quality of multi-attribute RF images from various perspectives, and construct the Multi-Attribute Retouched Face Database (MARFD) with multi-labels. Secondly, considering that retouching alters the facial morphology, we introduce a multi-task learning based No-Reference (NR) Image Quality Assessment (IQA) method, named MTNet. Specifically, to capture high-level semantic information associated with geometric changes, MTNet treats the alteration degree estimation of retouching attributes as auxiliary tasks for the main task (i.e., the overall quality prediction). In addition, inspired by the perceptual effects of viewing distance, MTNet utilizes a multi-scale data augmentation strategy during network training to help the network better understand the distortions. Experimental results on MARFD show that our MTNet correlates well with subjective ratings and outperforms 16 state-of-the-art NR-IQA methods.","PeriodicalId":13159,"journal":{"name":"IEEE Transactions on Broadcasting","volume":"70 2","pages":"570-583"},"PeriodicalIF":4.5,"publicationDate":"2024-03-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140316352","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：