Pub Date : 2024-07-03DOI: 10.1109/JETCAS.2024.3422371
Abhishek Sharma;Yanghyo Rod Kim
Conventional interconnects experience significant mechanical durability, mobility, and signal integrity challenges when dealing with moving parts or implementing extensive interconnect networks. As a result, they often hinder the performance of advanced autonomous and high-performance computing systems. This paper presents a fully rotatable and diagonally flexible ultra-short distance (≈ 1 mm) wireless interconnect. The proposed wireless interconnect comprises a 57-GHz transceiver integrated with a folded dipole antenna through wire bonding, enabling a flexible contactless connection. Here, two folded dipoles communicate in the Fresnel zone (radiative near-field), where we leverage the longitudinal electric fields to alleviate the polarization mismatch over the entire rotation angle. We have implemented a non-coherent on-off keying (OOK) modulation scheme and employed an automatic gain control (AGC) loop and offset canceling feedback loop to compensate for the transmission degradation and signal imbalance. The proposed system consumes 58.2 mW of power under a 1 V supply while transferring data at a rate of 10-Gb/s, achieving 5.82-pJ/bit energy efficiency.
{"title":"Energy-Efficient and Rotationally Adjustable Millimeter-Wave Wireless Interconnects","authors":"Abhishek Sharma;Yanghyo Rod Kim","doi":"10.1109/JETCAS.2024.3422371","DOIUrl":"10.1109/JETCAS.2024.3422371","url":null,"abstract":"Conventional interconnects experience significant mechanical durability, mobility, and signal integrity challenges when dealing with moving parts or implementing extensive interconnect networks. As a result, they often hinder the performance of advanced autonomous and high-performance computing systems. This paper presents a fully rotatable and diagonally flexible ultra-short distance (≈ 1 mm) wireless interconnect. The proposed wireless interconnect comprises a 57-GHz transceiver integrated with a folded dipole antenna through wire bonding, enabling a flexible contactless connection. Here, two folded dipoles communicate in the Fresnel zone (radiative near-field), where we leverage the longitudinal electric fields to alleviate the polarization mismatch over the entire rotation angle. We have implemented a non-coherent on-off keying (OOK) modulation scheme and employed an automatic gain control (AGC) loop and offset canceling feedback loop to compensate for the transmission degradation and signal imbalance. The proposed system consumes 58.2 mW of power under a 1 V supply while transferring data at a rate of 10-Gb/s, achieving 5.82-pJ/bit energy efficiency.","PeriodicalId":48827,"journal":{"name":"IEEE Journal on Emerging and Selected Topics in Circuits and Systems","volume":"14 3","pages":"551-562"},"PeriodicalIF":3.7,"publicationDate":"2024-07-03","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141548575","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"工程技术","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-07-01DOI: 10.1109/JETCAS.2024.3421553
Jicheon Kim;Chunmyung Park;Eunjae Hyun;Xuan Truong Nguyen;Hyuk-Jae Lee
Multi-chip-module (MCM) technology heralds a new era for scalable DNN inference systems, offering a cost-effective alternative to large-scale monolithic designs by lowering fabrication and design costs. Nevertheless, MCMs often incur resource and performance overheads due to inter-chip communication, which largely reduce a performance gain in a scaling-out system. To address these challenges, this paper introduces a highly-scalable DNN accelerator with a lightweight chip-to-chip adapter (C2CA) and a C2C-communication-aware scheduler. Our design employs a C2CA for inter-chip communication, which accurately illustrates an MCM system with a constrained C2C bandwidth, e.g., about 1/16, 1/8, or 1/4 of an on-chip bandwidth. We empirically reveal that the limited C2C bandwidth largely affects the overall performance gain of an MCM system. For example, compared with the one-core engine, a four-chip MCM system with a constrained C2C bandwidth only achieves $2.60times $ , $3.27times $ , $2.84times $ , and $2.74times $ performance gains on ResNet50, DarkNet19, MobileNetV1, and EfficientNetS, respectively. Mitigating the problem, we propose a novel C2C-communication-aware scheduler with forward and backward inter-layer scheduling. Specifically, our scheduler effectively utilizes a C2C bandwidth while a core is performing its own computation. To demonstrate the effectiveness and practicality of our concept, we modeled our design with Verilog HDL and implemented it on an FPGA board, i.e., Xilinx ZCU104. The experimental results demonstrate that the system shows significant throughput improvements compared to a single-chip configuration, yielding average enhancements of $1.87times $ and $3.43times $ for two-chip and four-chip configurations, respectively, on ResNet50, DarkNet19, MobileNetV1, and EfficientNetS.
{"title":"A Highly-Scalable Deep-Learning Accelerator With a Cost-Effective Chip-to-Chip Adapter and a C2C-Communication-Aware Scheduler","authors":"Jicheon Kim;Chunmyung Park;Eunjae Hyun;Xuan Truong Nguyen;Hyuk-Jae Lee","doi":"10.1109/JETCAS.2024.3421553","DOIUrl":"10.1109/JETCAS.2024.3421553","url":null,"abstract":"Multi-chip-module (MCM) technology heralds a new era for scalable DNN inference systems, offering a cost-effective alternative to large-scale monolithic designs by lowering fabrication and design costs. Nevertheless, MCMs often incur resource and performance overheads due to inter-chip communication, which largely reduce a performance gain in a scaling-out system. To address these challenges, this paper introduces a highly-scalable DNN accelerator with a lightweight chip-to-chip adapter (C2CA) and a C2C-communication-aware scheduler. Our design employs a C2CA for inter-chip communication, which accurately illustrates an MCM system with a constrained C2C bandwidth, e.g., about 1/16, 1/8, or 1/4 of an on-chip bandwidth. We empirically reveal that the limited C2C bandwidth largely affects the overall performance gain of an MCM system. For example, compared with the one-core engine, a four-chip MCM system with a constrained C2C bandwidth only achieves \u0000<inline-formula> <tex-math>$2.60times $ </tex-math></inline-formula>\u0000, \u0000<inline-formula> <tex-math>$3.27times $ </tex-math></inline-formula>\u0000, \u0000<inline-formula> <tex-math>$2.84times $ </tex-math></inline-formula>\u0000, and \u0000<inline-formula> <tex-math>$2.74times $ </tex-math></inline-formula>\u0000 performance gains on ResNet50, DarkNet19, MobileNetV1, and EfficientNetS, respectively. Mitigating the problem, we propose a novel C2C-communication-aware scheduler with forward and backward inter-layer scheduling. Specifically, our scheduler effectively utilizes a C2C bandwidth while a core is performing its own computation. To demonstrate the effectiveness and practicality of our concept, we modeled our design with Verilog HDL and implemented it on an FPGA board, i.e., Xilinx ZCU104. The experimental results demonstrate that the system shows significant throughput improvements compared to a single-chip configuration, yielding average enhancements of \u0000<inline-formula> <tex-math>$1.87times $ </tex-math></inline-formula>\u0000 and \u0000<inline-formula> <tex-math>$3.43times $ </tex-math></inline-formula>\u0000 for two-chip and four-chip configurations, respectively, on ResNet50, DarkNet19, MobileNetV1, and EfficientNetS.","PeriodicalId":48827,"journal":{"name":"IEEE Journal on Emerging and Selected Topics in Circuits and Systems","volume":"14 3","pages":"455-468"},"PeriodicalIF":3.7,"publicationDate":"2024-07-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141522295","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"工程技术","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-06-28DOI: 10.1109/JETCAS.2024.3420396
Zhen Gao;Ning Zhao;Guangdeng Zong;Xudong Zhao
Combining the use of the adaptive dynamic programming method and optimized backstepping strategy, this paper focuses on the secure consensus problem for constrained nonlinear multi-agent systems (MASs) subject to denial-of-service (DoS) attacks and input delay. Since network channels between some agents often suffer from intrusions by attackers during data transmission, we consider information transfers in both attack-sleep and attack-active scenarios, and construct a novel distributed observer with a switched mechanism to estimate the leader’s state information. In order to optimize system performances while ensuring that the system states do not exceed constraint sets, a new performance index function and a tan-type barrier Lyapunov function (BLF) are introduced. Besides, by employing the Pade approximation and an intermediate variable, the effect of input delay is removed. As a consequence, the proposed optimal control can smoothly steer the nonlinear MASs to realize the followers-leader consensus tracking goal, and all system states are consistently constrained within their compact sets. Finally, simulation results verify the effectiveness of this control scheme.
{"title":"Secure Consensus Control for Constrained Multi-Agent Systems Against Intermittent Denial-of-Service Attacks: An Adaptive Dynamic Programming Method","authors":"Zhen Gao;Ning Zhao;Guangdeng Zong;Xudong Zhao","doi":"10.1109/JETCAS.2024.3420396","DOIUrl":"10.1109/JETCAS.2024.3420396","url":null,"abstract":"Combining the use of the adaptive dynamic programming method and optimized backstepping strategy, this paper focuses on the secure consensus problem for constrained nonlinear multi-agent systems (MASs) subject to denial-of-service (DoS) attacks and input delay. Since network channels between some agents often suffer from intrusions by attackers during data transmission, we consider information transfers in both attack-sleep and attack-active scenarios, and construct a novel distributed observer with a switched mechanism to estimate the leader’s state information. In order to optimize system performances while ensuring that the system states do not exceed constraint sets, a new performance index function and a tan-type barrier Lyapunov function (BLF) are introduced. Besides, by employing the Pade approximation and an intermediate variable, the effect of input delay is removed. As a consequence, the proposed optimal control can smoothly steer the nonlinear MASs to realize the followers-leader consensus tracking goal, and all system states are consistently constrained within their compact sets. Finally, simulation results verify the effectiveness of this control scheme.","PeriodicalId":48827,"journal":{"name":"IEEE Journal on Emerging and Selected Topics in Circuits and Systems","volume":"14 4","pages":"705-716"},"PeriodicalIF":3.7,"publicationDate":"2024-06-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10577119","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141508148","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"工程技术","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-06-01DOI: 10.1109/JETCAS.2024.3417549
{"title":"IEEE Journal on Emerging and Selected Topics in Circuits and Systems information for authors","authors":"","doi":"10.1109/JETCAS.2024.3417549","DOIUrl":"https://doi.org/10.1109/JETCAS.2024.3417549","url":null,"abstract":"","PeriodicalId":48827,"journal":{"name":"IEEE Journal on Emerging and Selected Topics in Circuits and Systems","volume":"14 2","pages":"348-348"},"PeriodicalIF":3.7,"publicationDate":"2024-06-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10579095","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141494812","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"工程技术","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-06-01DOI: 10.1109/JETCAS.2024.3405090
{"title":"IEEE Journal on Emerging and Selected Topics in Circuits and Systems Publication Information","authors":"","doi":"10.1109/JETCAS.2024.3405090","DOIUrl":"https://doi.org/10.1109/JETCAS.2024.3405090","url":null,"abstract":"","PeriodicalId":48827,"journal":{"name":"IEEE Journal on Emerging and Selected Topics in Circuits and Systems","volume":"14 2","pages":"C2-C2"},"PeriodicalIF":3.7,"publicationDate":"2024-06-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10579073","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141495163","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"工程技术","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-06-01DOI: 10.1109/JETCAS.2024.3405094
{"title":"IEEE Circuits and Systems Society","authors":"","doi":"10.1109/JETCAS.2024.3405094","DOIUrl":"https://doi.org/10.1109/JETCAS.2024.3405094","url":null,"abstract":"","PeriodicalId":48827,"journal":{"name":"IEEE Journal on Emerging and Selected Topics in Circuits and Systems","volume":"14 2","pages":"C3-C3"},"PeriodicalIF":3.7,"publicationDate":"2024-06-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10579094","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141495247","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"工程技术","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-06-01DOI: 10.1109/JETCAS.2024.3403318
Zhibo Chen;Heming Sun;Li Zhang;Fan Zhang
This special issue of IEEE Journal on Emerging and Selected Topics in Circuits and Systems (JETCAS) is dedicated to demonstrating the latest developments in algorithms, implementations, and applications related to visual signal coding and processing with generative models. In recent years, generative models have emerged as one of the most significant and rapidly developing areas of research in artificial intelligence. They have proved to be an important instrument for advancing research in AI-based visual signal coding and processing. For instance, the variational autoencoder (VAE) has been used as a fundamental framework for end-to-end learned image coding, the autoregressive (AR) model has been extensively studied for efficient entropy coding, and the generative adversarial network (GAN) has been utilized frequently to enhance the subjective quality of coding schemes. Meanwhile, generative models have also been explored in various visual signal processing tasks, including quality assessment, restoration, enhancement, editing, and interpolation.
本期《电气和电子工程师学会电路与系统新兴选题期刊》（IEEE Journal on Emerging and Selected Topics in Circuits and Systems，JETCAS）特刊致力于展示与生成模型视觉信号编码和处理相关的算法、实现和应用方面的最新进展。近年来，生成模型已成为人工智能领域最重要、发展最迅速的研究领域之一。事实证明，它们是推动基于人工智能的视觉信号编码和处理研究的重要工具。例如，变分自动编码器（VAE）已被用作端到端学习图像编码的基本框架，自回归（AR）模型已被广泛研究用于高效熵编码，生成对抗网络（GAN）已被频繁用于提高编码方案的主观质量。同时，生成模型还在各种视觉信号处理任务中得到了应用，包括质量评估、修复、增强、编辑和插值。
{"title":"Guest Editorial Advances in Generative Visual Signal Coding and Processing","authors":"Zhibo Chen;Heming Sun;Li Zhang;Fan Zhang","doi":"10.1109/JETCAS.2024.3403318","DOIUrl":"https://doi.org/10.1109/JETCAS.2024.3403318","url":null,"abstract":"This special issue of IEEE Journal on Emerging and Selected Topics in Circuits and Systems (JETCAS) is dedicated to demonstrating the latest developments in algorithms, implementations, and applications related to visual signal coding and processing with generative models. In recent years, generative models have emerged as one of the most significant and rapidly developing areas of research in artificial intelligence. They have proved to be an important instrument for advancing research in AI-based visual signal coding and processing. For instance, the variational autoencoder (VAE) has been used as a fundamental framework for end-to-end learned image coding, the autoregressive (AR) model has been extensively studied for efficient entropy coding, and the generative adversarial network (GAN) has been utilized frequently to enhance the subjective quality of coding schemes. Meanwhile, generative models have also been explored in various visual signal processing tasks, including quality assessment, restoration, enhancement, editing, and interpolation.","PeriodicalId":48827,"journal":{"name":"IEEE Journal on Emerging and Selected Topics in Circuits and Systems","volume":"14 2","pages":"145-148"},"PeriodicalIF":3.7,"publicationDate":"2024-06-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10579096","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141495162","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"工程技术","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-04-30DOI: 10.1109/JETCAS.2024.3395418
Issa Khalifeh;Luka Murn;Ebroul Izquierdo
Video frame interpolation synthesises a new frame from existing frames. Several approaches have been devised to handle this core computer vision problem. Kernel-based approaches use an encoder-decoder architecture to extract features from the inputs and generate weights for a local separable convolution operation which is used to warp the input frames. The warped inputs are then combined to obtain the final interpolated frame. The ease of implementation of such an approach and favourable performance have enabled it to become a popular method in the field of interpolation. One downside, however, is that the encoder-decoder feature extractor is large and uses a lot of parameters. We propose a Multi-Encoder Method for Parameter Reduction (MEMPR) that can significantly reduce parameters by up to 85% whilst maintaining a similar level of performance. This is achieved by leveraging multiple encoders to focus on different aspects of the input. The approach can also be used to improve the performance of kernel-based models in a parameter-effective manner. To encourage the adoption of such an approach in potential future kernel-based methods, the approach is designed to be modular, intuitive and easy to implement. It is implemented on some of the most impactful kernel-based works such as SepConvNet, AdaCoFNet and EDSC. Extensive experiments on datasets with varying ranges of motion highlight the effectiveness of the MEMPR approach and its generalisability to different convolutional backbones and kernel-based operators.
{"title":"Parameter Reduction of Kernel-Based Video Frame Interpolation Methods Using Multiple Encoders","authors":"Issa Khalifeh;Luka Murn;Ebroul Izquierdo","doi":"10.1109/JETCAS.2024.3395418","DOIUrl":"10.1109/JETCAS.2024.3395418","url":null,"abstract":"Video frame interpolation synthesises a new frame from existing frames. Several approaches have been devised to handle this core computer vision problem. Kernel-based approaches use an encoder-decoder architecture to extract features from the inputs and generate weights for a local separable convolution operation which is used to warp the input frames. The warped inputs are then combined to obtain the final interpolated frame. The ease of implementation of such an approach and favourable performance have enabled it to become a popular method in the field of interpolation. One downside, however, is that the encoder-decoder feature extractor is large and uses a lot of parameters. We propose a Multi-Encoder Method for Parameter Reduction (MEMPR) that can significantly reduce parameters by up to 85% whilst maintaining a similar level of performance. This is achieved by leveraging multiple encoders to focus on different aspects of the input. The approach can also be used to improve the performance of kernel-based models in a parameter-effective manner. To encourage the adoption of such an approach in potential future kernel-based methods, the approach is designed to be modular, intuitive and easy to implement. It is implemented on some of the most impactful kernel-based works such as SepConvNet, AdaCoFNet and EDSC. Extensive experiments on datasets with varying ranges of motion highlight the effectiveness of the MEMPR approach and its generalisability to different convolutional backbones and kernel-based operators.","PeriodicalId":48827,"journal":{"name":"IEEE Journal on Emerging and Selected Topics in Circuits and Systems","volume":"14 2","pages":"245-260"},"PeriodicalIF":3.7,"publicationDate":"2024-04-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10510388","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140826670","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"工程技术","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-04-29DOI: 10.1109/JETCAS.2024.3394495
Jiang Zhu;Van Kwan Zhi Koh;Zhiping Lin;Bihan Wen
Despite significant strides in deep single image super-resolution (SISR), the development of robust guided depth image super-resolution (GDSR) techniques presents a notable challenge. Effective GDSR methods must not only exploit the properties of the target image but also integrate complementary information from the guidance image. The state-of-the-art in guided image super-resolution has been dominated by convolutional neural network (CNN) based methods, which leverage CNN as their architecture. However, CNN has limitations in capturing global information effectively, and their traditional regression training techniques can sometimes lead to challenges in the precise generating of high-frequency details, unlike transformers that have shown remarkable success in deep learning through the self-attention mechanism. Drawing inspiration from the transformative impact of transformers in both language and vision applications, we propose a Transformer-based Multi-modal Generative Adversarial Network dubbed TM-GAN. TM-GAN is designed to effectively process and integrate multi-modal data, leveraging the global contextual understanding and detailed feature extraction capabilities of transformers within a GAN architecture for GDSR, aiming to effectively integrate and utilize multi-modal data sources. Experimental evaluations of TM-GAN on a variety of RGB-D datasets demonstrate its superiority over the state-of-the-art methods, showcasing its effectiveness in leveraging transformer-based techniques for GDSR.
{"title":"TM-GAN: A Transformer-Based Multi-Modal Generative Adversarial Network for Guided Depth Image Super-Resolution","authors":"Jiang Zhu;Van Kwan Zhi Koh;Zhiping Lin;Bihan Wen","doi":"10.1109/JETCAS.2024.3394495","DOIUrl":"10.1109/JETCAS.2024.3394495","url":null,"abstract":"Despite significant strides in deep single image super-resolution (SISR), the development of robust guided depth image super-resolution (GDSR) techniques presents a notable challenge. Effective GDSR methods must not only exploit the properties of the target image but also integrate complementary information from the guidance image. The state-of-the-art in guided image super-resolution has been dominated by convolutional neural network (CNN) based methods, which leverage CNN as their architecture. However, CNN has limitations in capturing global information effectively, and their traditional regression training techniques can sometimes lead to challenges in the precise generating of high-frequency details, unlike transformers that have shown remarkable success in deep learning through the self-attention mechanism. Drawing inspiration from the transformative impact of transformers in both language and vision applications, we propose a Transformer-based Multi-modal Generative Adversarial Network dubbed TM-GAN. TM-GAN is designed to effectively process and integrate multi-modal data, leveraging the global contextual understanding and detailed feature extraction capabilities of transformers within a GAN architecture for GDSR, aiming to effectively integrate and utilize multi-modal data sources. Experimental evaluations of TM-GAN on a variety of RGB-D datasets demonstrate its superiority over the state-of-the-art methods, showcasing its effectiveness in leveraging transformer-based techniques for GDSR.","PeriodicalId":48827,"journal":{"name":"IEEE Journal on Emerging and Selected Topics in Circuits and Systems","volume":"14 2","pages":"261-274"},"PeriodicalIF":3.7,"publicationDate":"2024-04-29","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140826640","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"工程技术","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-04-29DOI: 10.1109/JETCAS.2024.3394878
Ruolei Ji;Lina J. Karam
Compressed-domain visual task schemes, where visual processing or computer vision are directly performed on the compressed-domain representations, were shown to achieve a higher computational efficiency during training and deployment by avoiding the need to decode the compressed visual information while resulting in a competitive or even better performance as compared to corresponding spatial-domain visual tasks. This work is concerned with learning-based compressed-domain image classification, where the image classification is performed directly on compressed-domain representations, also known as latent representations, that are obtained using a learning-based visual encoder. In this paper, a compressed-domain Vision Transformer (cViT) is proposed to perform image classification in the learning-based compressed-domain. For this purpose, the Vision Transformer (ViT) architecture is adopted and modified to perform classification directly in the compressed-domain. As part of this work, a novel feature patch embedding is introduced leveraging the within- and cross-channel information in the compressed-domain. Also, an adaptation training strategy is designed to adopt the weights from the pre-trained spatial-domain ViT and adapt these to the compressed-domain classification task. Furthermore, the pre-trained ViT weights are utilized through interpolation for position embedding initialization to further improve the performance of cViT. The experimental results show that the proposed cViT outperforms the existing compressed-domain classification networks in terms of Top-1 and Top-5 classification accuracies. Moreover, the proposed cViT can yield competitive classification accuracies with a significantly higher computational efficiency as compared to pixel-domain approaches.
在压缩域视觉任务方案中，视觉处理或计算机视觉直接在压缩域表征上执行，通过避免对压缩视觉信息进行解码，在训练和部署过程中实现了更高的计算效率，同时与相应的空间域视觉任务相比，具有竞争力甚至更好的性能。这项工作关注的是基于学习的压缩域图像分类，即直接在压缩域表征（也称为潜在表征）上执行图像分类，这些表征是使用基于学习的视觉编码器获得的。本文提出了一种压缩域视觉变换器（cViT），用于在基于学习的压缩域中执行图像分类。为此，本文采用并修改了视觉变换器（ViT）架构，以便直接在压缩域中执行分类。作为这项工作的一部分，我们引入了一种新颖的特征补丁嵌入方法，利用压缩域中的内部和跨通道信息。此外，还设计了一种适应性训练策略，采用预先训练好的空间域 ViT 的权重，并将其适应于压缩域分类任务。此外，预训练的 ViT 权重通过插值法用于位置嵌入初始化，以进一步提高 cViT 的性能。实验结果表明，所提出的 cViT 在分类精度 Top-1 和 Top-5 方面优于现有的压缩域分类网络。此外，与像素域方法相比，所提出的 cViT 能以更高的计算效率获得有竞争力的分类精度。
{"title":"Compressed-Domain Vision Transformer for Image Classification","authors":"Ruolei Ji;Lina J. Karam","doi":"10.1109/JETCAS.2024.3394878","DOIUrl":"10.1109/JETCAS.2024.3394878","url":null,"abstract":"Compressed-domain visual task schemes, where visual processing or computer vision are directly performed on the compressed-domain representations, were shown to achieve a higher computational efficiency during training and deployment by avoiding the need to decode the compressed visual information while resulting in a competitive or even better performance as compared to corresponding spatial-domain visual tasks. This work is concerned with learning-based compressed-domain image classification, where the image classification is performed directly on compressed-domain representations, also known as latent representations, that are obtained using a learning-based visual encoder. In this paper, a compressed-domain Vision Transformer (cViT) is proposed to perform image classification in the learning-based compressed-domain. For this purpose, the Vision Transformer (ViT) architecture is adopted and modified to perform classification directly in the compressed-domain. As part of this work, a novel feature patch embedding is introduced leveraging the within- and cross-channel information in the compressed-domain. Also, an adaptation training strategy is designed to adopt the weights from the pre-trained spatial-domain ViT and adapt these to the compressed-domain classification task. Furthermore, the pre-trained ViT weights are utilized through interpolation for position embedding initialization to further improve the performance of cViT. The experimental results show that the proposed cViT outperforms the existing compressed-domain classification networks in terms of Top-1 and Top-5 classification accuracies. Moreover, the proposed cViT can yield competitive classification accuracies with a significantly higher computational efficiency as compared to pixel-domain approaches.","PeriodicalId":48827,"journal":{"name":"IEEE Journal on Emerging and Selected Topics in Circuits and Systems","volume":"14 2","pages":"299-310"},"PeriodicalIF":3.7,"publicationDate":"2024-04-29","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140826671","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"工程技术","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：