Fusion of a panchromatic (PAN) image and corresponding multispectral (MS) image is also known as pansharpening, which aims to combine abundant spatial details of PAN and spectral information of MS images. Due to the absence of high-resolution MS images, available deep-learning-based methods usually follow the paradigm of training at reduced resolution and testing at both reduced and full resolution. When taking original MS and PAN images as inputs, they always obtain sub-optimal results due to the scale variation. In this paper, we propose to explore the self-supervised representation for pansharpening by designing a cross-predictive diffusion model, named CrossDiff. It has two-stage training. In the first stage, we introduce a cross-predictive pretext task to pre-train the UNet structure based on conditional Denoising Diffusion Probabilistic Model (DDPM). While in the second stage, the encoders of the UNets are frozen to directly extract spatial and spectral features from PAN and MS images, and only the fusion head is trained to adapt for pansharpening task. Extensive experiments show the effectiveness and superiority of the proposed model compared with state-of-the-art supervised and unsupervised methods. Besides, the cross-sensor experiments also verify the generalization ability of proposed self-supervised representation learners for other satellite datasets. Code is available at https://github.com/codgodtao/CrossDiff .
全色（PAN）图像与相应的多光谱（MS）图像的融合也被称为泛锐化，其目的是将全色（PAN）图像的丰富空间细节与多光谱（MS）图像的光谱信息结合起来。由于缺乏高分辨率的 MS 图像，现有的基于深度学习的方法通常采用降低分辨率进行训练、同时在降低分辨率和全分辨率下进行测试的模式。在将原始 MS 和 PAN 图像作为输入时，由于尺度的变化，它们总能获得次优结果。在本文中，我们建议通过设计一种名为 CrossDiff 的交叉预测扩散模型来探索用于平差处理的自监督表示方法。它有两个阶段的训练。在第一阶段，我们基于条件去噪扩散概率模型（DDPM），引入交叉预测借口任务来预训练 UNet 结构。而在第二阶段，UNet 的编码器被冻结，直接从 PAN 和 MS 图像中提取空间和光谱特征，只训练融合头以适应泛锐化任务。大量实验表明，与最先进的有监督和无监督方法相比，所提出的模型更加有效和优越。此外，跨传感器实验也验证了所提出的自监督表示学习器对其他卫星数据集的泛化能力。代码见 https://github.com/codgodtao/CrossDiff。
{"title":"CrossDiff: Exploring Self-SupervisedRepresentation of Pansharpening via Cross-Predictive Diffusion Model","authors":"Yinghui Xing;Litao Qu;Shizhou Zhang;Kai Zhang;Yanning Zhang;Lorenzo Bruzzone","doi":"10.1109/TIP.2024.3461476","DOIUrl":"10.1109/TIP.2024.3461476","url":null,"abstract":"Fusion of a panchromatic (PAN) image and corresponding multispectral (MS) image is also known as pansharpening, which aims to combine abundant spatial details of PAN and spectral information of MS images. Due to the absence of high-resolution MS images, available deep-learning-based methods usually follow the paradigm of training at reduced resolution and testing at both reduced and full resolution. When taking original MS and PAN images as inputs, they always obtain sub-optimal results due to the scale variation. In this paper, we propose to explore the self-supervised representation for pansharpening by designing a cross-predictive diffusion model, named CrossDiff. It has two-stage training. In the first stage, we introduce a cross-predictive pretext task to pre-train the UNet structure based on conditional Denoising Diffusion Probabilistic Model (DDPM). While in the second stage, the encoders of the UNets are frozen to directly extract spatial and spectral features from PAN and MS images, and only the fusion head is trained to adapt for pansharpening task. Extensive experiments show the effectiveness and superiority of the proposed model compared with state-of-the-art supervised and unsupervised methods. Besides, the cross-sensor experiments also verify the generalization ability of proposed self-supervised representation learners for other satellite datasets. Code is available at \u0000<uri>https://github.com/codgodtao/CrossDiff</uri>\u0000.","PeriodicalId":94032,"journal":{"name":"IEEE transactions on image processing : a publication of the IEEE Signal Processing Society","volume":"33 ","pages":"5496-5509"},"PeriodicalIF":0.0,"publicationDate":"2024-09-20","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142275357","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Most existing few-shot learning (FSL) methods require a large amount of labeled data in meta-training, which is a major limit. To reduce the requirement of labels, a semi-supervised meta-training (SSMT) setting has been proposed for FSL, which includes only a few labeled samples and numbers of unlabeled samples in base classes. However, existing methods under this setting require class-aware sample selection from the unlabeled set, which violates the assumption of unlabeled set. In this paper, we propose a practical semi-supervised meta-training setting with truly unlabeled data to facilitate the applications of FSL in realistic scenarios. To better utilize both the labeled and truly unlabeled data, we propose a simple and effective meta-training framework, called pseudo-labeling based meta-learning (PLML). Firstly, we train a classifier via common semi-supervised learning (SSL) and use it to obtain the pseudo-labels of unlabeled data. Then we build few-shot tasks from labeled and pseudo-labeled data and design a novel finetuning method with feature smoothing and noise suppression to better learn the FSL model from noise labels. Surprisingly, through extensive experiments across two FSL datasets, we find that this simple meta-training framework effectively prevents the performance degradation of various FSL models under limited labeled data, and also significantly outperforms the representative SSMT models. Besides, benefiting from meta-training, our method also improves several representative SSL algorithms as well. We provide the training code and usage examples at https://github.com/ouyangtianran/PLML .
{"title":"Pseudo-Labeling Based Practical Semi-Supervised Meta-Training for Few-Shot Learning","authors":"Xingping Dong;Tianran Ouyang;Shengcai Liao;Bo Du;Ling Shao","doi":"10.1109/TIP.2024.3461472","DOIUrl":"10.1109/TIP.2024.3461472","url":null,"abstract":"Most existing few-shot learning (FSL) methods require a large amount of labeled data in meta-training, which is a major limit. To reduce the requirement of labels, a semi-supervised meta-training (SSMT) setting has been proposed for FSL, which includes only a few labeled samples and numbers of unlabeled samples in base classes. However, existing methods under this setting require class-aware sample selection from the unlabeled set, which violates the assumption of unlabeled set. In this paper, we propose a practical semi-supervised meta-training setting with truly unlabeled data to facilitate the applications of FSL in realistic scenarios. To better utilize both the labeled and truly unlabeled data, we propose a simple and effective meta-training framework, called pseudo-labeling based meta-learning (PLML). Firstly, we train a classifier via common semi-supervised learning (SSL) and use it to obtain the pseudo-labels of unlabeled data. Then we build few-shot tasks from labeled and pseudo-labeled data and design a novel finetuning method with feature smoothing and noise suppression to better learn the FSL model from noise labels. Surprisingly, through extensive experiments across two FSL datasets, we find that this simple meta-training framework effectively prevents the performance degradation of various FSL models under limited labeled data, and also significantly outperforms the representative SSMT models. Besides, benefiting from meta-training, our method also improves several representative SSL algorithms as well. We provide the training code and usage examples at \u0000<uri>https://github.com/ouyangtianran/PLML</uri>\u0000.","PeriodicalId":94032,"journal":{"name":"IEEE transactions on image processing : a publication of the IEEE Signal Processing Society","volume":"33 ","pages":"5663-5675"},"PeriodicalIF":0.0,"publicationDate":"2024-09-20","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142275354","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-09-19DOI: 10.1109/TIP.2024.3460470
Qian Hu;Xinya Wang;Junjun Jiang;Xiao-Ping Zhang;Jiayi Ma
In recent years, many single hyperspectral image super-resolution methods have emerged to enhance the spatial resolution of hyperspectral images without hardware modification. However, existing methods typically face two significant challenges. First, they struggle to handle the high-dimensional nature of hyperspectral data, which often results in high computational complexity and inefficient information utilization. Second, they have not fully leveraged the abundant spectral information in hyperspectral images. To address these challenges, we propose a novel hyperspectral super-resolution network named SNLSR, which transfers the super-resolution problem into the abundance domain. Our SNLSR leverages a spatial preserve decomposition network to estimate the abundance representations of the input hyperspectral image. Notably, the network acknowledges and utilizes the commonly overlooked spatial correlations of hyperspectral images, leading to better reconstruction performance. Then, the estimated low-resolution abundance is super-resolved through a spatial spectral attention network, where the informative features from both spatial and spectral domains are fully exploited. Considering that the hyperspectral image is highly spectrally correlated, we customize a spectral-wise non-local attention module to mine similar pixels along spectral dimension for high-frequency detail recovery. Extensive experiments demonstrate the superiority of our method over other state-of-the-art methods both visually and metrically. Our code is publicly available at https://github.com/HuQ1an/SNLSR .
{"title":"Exploring the Spectral Prior for Hyperspectral Image Super-Resolution","authors":"Qian Hu;Xinya Wang;Junjun Jiang;Xiao-Ping Zhang;Jiayi Ma","doi":"10.1109/TIP.2024.3460470","DOIUrl":"10.1109/TIP.2024.3460470","url":null,"abstract":"In recent years, many single hyperspectral image super-resolution methods have emerged to enhance the spatial resolution of hyperspectral images without hardware modification. However, existing methods typically face two significant challenges. First, they struggle to handle the high-dimensional nature of hyperspectral data, which often results in high computational complexity and inefficient information utilization. Second, they have not fully leveraged the abundant spectral information in hyperspectral images. To address these challenges, we propose a novel hyperspectral super-resolution network named SNLSR, which transfers the super-resolution problem into the abundance domain. Our SNLSR leverages a spatial preserve decomposition network to estimate the abundance representations of the input hyperspectral image. Notably, the network acknowledges and utilizes the commonly overlooked spatial correlations of hyperspectral images, leading to better reconstruction performance. Then, the estimated low-resolution abundance is super-resolved through a spatial spectral attention network, where the informative features from both spatial and spectral domains are fully exploited. Considering that the hyperspectral image is highly spectrally correlated, we customize a spectral-wise non-local attention module to mine similar pixels along spectral dimension for high-frequency detail recovery. Extensive experiments demonstrate the superiority of our method over other state-of-the-art methods both visually and metrically. Our code is publicly available at \u0000<uri>https://github.com/HuQ1an/SNLSR</uri>\u0000.","PeriodicalId":94032,"journal":{"name":"IEEE transactions on image processing : a publication of the IEEE Signal Processing Society","volume":"33 ","pages":"5260-5272"},"PeriodicalIF":0.0,"publicationDate":"2024-09-19","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142273380","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-09-18DOI: 10.1109/TIP.2024.3459651
Yalan Qin;Chuan Qin;Xinpeng Zhang;Guorui Feng
Multi-view clustering usually attempts to improve the final performance by integrating graph structure information from different views and methods based on anchor are presented to reduce the computation cost for datasets with large scales. Despite significant progress, these methods pay few attentions to ensuring that the cluster structure correspondence between anchor graph and partition is built on multi-view datasets. Besides, they ignore to discover the anchor graph depicting the shared cluster assignment across views under the orthogonal constraint on actual bases in factorization. In this paper, we propose a novel Dual consensus Anchor Learning for Fast multi-view clustering (DALF) method, where the cluster structure correspondence between anchor graph and partition is guaranteed on multi-view datasets with large scales. It jointly learns anchors, constructs anchor graph and performs partition under a unified framework with the rank constraint imposed on the built Laplacian graph and the orthogonal constraint on the centroid representation. DALF simultaneously focuses on the cluster structure in the anchor graph and partition. The final cluster structure is simultaneously shown in the anchor graph and partition. We introduce the orthogonal constraint on the centroid representation in anchor graph factorization and the cluster assignment is directly constructed, where the cluster structure is shown in the partition. We present an iterative algorithm for solving the formulated problem. Extensive experiments demonstrate the effectiveness and efficiency of DALF on different multi-view datasets compared with other methods.
多视图聚类通常试图通过整合来自不同视图的图结构信息来提高最终性能，并提出了基于锚的方法来降低大规模数据集的计算成本。尽管取得了重大进展，但这些方法很少关注如何确保在多视图数据集上建立锚图与分区之间的聚类结构对应关系。此外，这些方法忽视了在因式分解的实际基数正交约束下发现描述跨视图共享聚类分配的锚图。在本文中，我们提出了一种新颖的用于快速多视图聚类的双共识锚点学习（Dual consensus Anchor Learning for Fast multi-view clustering，DALF）方法，在这种方法中，锚点图和分区之间的聚类结构对应关系在大尺度多视图数据集上得到了保证。它在统一的框架下联合学习锚点、构建锚点图并执行分区，对构建的拉普拉斯图施加秩约束，对中心点表示施加正交约束。DALF 同时关注锚图和分区中的聚类结构。最终的聚类结构同时显示在锚图和分区中。我们在锚图因式分解中引入了对中心点表示的正交约束，并直接构建了聚类分配，聚类结构显示在分区中。我们提出了解决所提问题的迭代算法。大量实验证明，与其他方法相比，DALF 在不同多视图数据集上的有效性和高效性。
{"title":"Dual Consensus Anchor Learning for Fast Multi-View Clustering","authors":"Yalan Qin;Chuan Qin;Xinpeng Zhang;Guorui Feng","doi":"10.1109/TIP.2024.3459651","DOIUrl":"10.1109/TIP.2024.3459651","url":null,"abstract":"Multi-view clustering usually attempts to improve the final performance by integrating graph structure information from different views and methods based on anchor are presented to reduce the computation cost for datasets with large scales. Despite significant progress, these methods pay few attentions to ensuring that the cluster structure correspondence between anchor graph and partition is built on multi-view datasets. Besides, they ignore to discover the anchor graph depicting the shared cluster assignment across views under the orthogonal constraint on actual bases in factorization. In this paper, we propose a novel Dual consensus Anchor Learning for Fast multi-view clustering (DALF) method, where the cluster structure correspondence between anchor graph and partition is guaranteed on multi-view datasets with large scales. It jointly learns anchors, constructs anchor graph and performs partition under a unified framework with the rank constraint imposed on the built Laplacian graph and the orthogonal constraint on the centroid representation. DALF simultaneously focuses on the cluster structure in the anchor graph and partition. The final cluster structure is simultaneously shown in the anchor graph and partition. We introduce the orthogonal constraint on the centroid representation in anchor graph factorization and the cluster assignment is directly constructed, where the cluster structure is shown in the partition. We present an iterative algorithm for solving the formulated problem. Extensive experiments demonstrate the effectiveness and efficiency of DALF on different multi-view datasets compared with other methods.","PeriodicalId":94032,"journal":{"name":"IEEE transactions on image processing : a publication of the IEEE Signal Processing Society","volume":"33 ","pages":"5298-5311"},"PeriodicalIF":0.0,"publicationDate":"2024-09-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142245414","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-09-18DOI: 10.1109/TIP.2024.3459788
Hongbo Sun;Xiangteng He;Jinglin Xu;Yuxin Peng
Fine-grained visual categorization (FGVC) aims to distinguish visual objects from multiple subcategories of the coarse-grained category. Subtle inter-class differences among various subcategories make the FGVC task more challenging. Existing methods primarily focus on learning salient visual patterns while ignoring how to capture the object’s internal structure, causing difficulty in obtaining complete discriminative regions within the object to limit FGVC performance. To address the above issue, we propose a Structure Information Mining and Object-aware Feature Enhancement (SIM-OFE) method for fine-grained visual categorization, which explores the visual object’s internal structure composition and appearance traits. Concretely, we first propose a simple yet effective hybrid perception attention module for locating visual objects based on global-scope and local-scope significance analyses. Then, a structure information mining module is proposed to model the distribution and context relation of critical regions within the object, highlighting the whole object and discriminative regions for distinguishing subtle differences. Finally, an object-aware feature enhancement module is proposed to combine global-scope and local-scope discriminative features in an attentive coupling way for powerful visual representations in fine-grained recognition. Extensive experiments on three FGVC benchmark datasets demonstrate that our proposed SIM-OFE method can achieve state-of-the-art performance.
{"title":"SIM-OFE: Structure Information Mining and Object-Aware Feature Enhancement for Fine-Grained Visual Categorization","authors":"Hongbo Sun;Xiangteng He;Jinglin Xu;Yuxin Peng","doi":"10.1109/TIP.2024.3459788","DOIUrl":"10.1109/TIP.2024.3459788","url":null,"abstract":"Fine-grained visual categorization (FGVC) aims to distinguish visual objects from multiple subcategories of the coarse-grained category. Subtle inter-class differences among various subcategories make the FGVC task more challenging. Existing methods primarily focus on learning salient visual patterns while ignoring how to capture the object’s internal structure, causing difficulty in obtaining complete discriminative regions within the object to limit FGVC performance. To address the above issue, we propose a Structure Information Mining and Object-aware Feature Enhancement (SIM-OFE) method for fine-grained visual categorization, which explores the visual object’s internal structure composition and appearance traits. Concretely, we first propose a simple yet effective hybrid perception attention module for locating visual objects based on global-scope and local-scope significance analyses. Then, a structure information mining module is proposed to model the distribution and context relation of critical regions within the object, highlighting the whole object and discriminative regions for distinguishing subtle differences. Finally, an object-aware feature enhancement module is proposed to combine global-scope and local-scope discriminative features in an attentive coupling way for powerful visual representations in fine-grained recognition. Extensive experiments on three FGVC benchmark datasets demonstrate that our proposed SIM-OFE method can achieve state-of-the-art performance.","PeriodicalId":94032,"journal":{"name":"IEEE transactions on image processing : a publication of the IEEE Signal Processing Society","volume":"33 ","pages":"5312-5326"},"PeriodicalIF":0.0,"publicationDate":"2024-09-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142245412","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-09-18DOI: 10.1109/TIP.2024.3459626
Junyu Gao;Xinhong Ma;Changsheng Xu
Despite the great progress of unsupervised domain adaptation (UDA) with the deep neural networks, current UDA models are opaque and cannot provide promising explanations, limiting their applications in the scenarios that require safe and controllable model decisions. At present, a surge of work focuses on designing deep interpretable methods with adequate data annotations and only a few methods consider the distributional shift problem. Most existing interpretable UDA methods are post-hoc ones, which cannot facilitate the model learning process for performance enhancement. In this paper, we propose an inherently interpretable method, named Transferable Conceptual Prototype Learning (TCPL), which could simultaneously interpret and improve the processes of knowledge transfer and decision-making in UDA. To achieve this goal, we design a hierarchically prototypical module that transfers categorical basic concepts from the source domain to the target domain and learns domain-shared prototypes for explaining the underlying reasoning process. With the learned transferable prototypes, a self-predictive consistent pseudo-label strategy that fuses confidence, predictions, and prototype information, is designed for selecting suitable target samples for pseudo annotations and gradually narrowing down the domain gap. Comprehensive experiments show that the proposed method can not only provide effective and intuitive explanations but also outperform previous state-of-the-arts. Code is available at https://drive.google.com/file/d/1b1EHFghiF1ExD-Cn1HYg75VutfkXWp60/view?usp=sharing .
{"title":"Learning Transferable Conceptual Prototypes for Interpretable Unsupervised Domain Adaptation","authors":"Junyu Gao;Xinhong Ma;Changsheng Xu","doi":"10.1109/TIP.2024.3459626","DOIUrl":"10.1109/TIP.2024.3459626","url":null,"abstract":"Despite the great progress of unsupervised domain adaptation (UDA) with the deep neural networks, current UDA models are opaque and cannot provide promising explanations, limiting their applications in the scenarios that require safe and controllable model decisions. At present, a surge of work focuses on designing deep interpretable methods with adequate data annotations and only a few methods consider the distributional shift problem. Most existing interpretable UDA methods are post-hoc ones, which cannot facilitate the model learning process for performance enhancement. In this paper, we propose an inherently interpretable method, named Transferable Conceptual Prototype Learning (TCPL), which could simultaneously interpret and improve the processes of knowledge transfer and decision-making in UDA. To achieve this goal, we design a hierarchically prototypical module that transfers categorical basic concepts from the source domain to the target domain and learns domain-shared prototypes for explaining the underlying reasoning process. With the learned transferable prototypes, a self-predictive consistent pseudo-label strategy that fuses confidence, predictions, and prototype information, is designed for selecting suitable target samples for pseudo annotations and gradually narrowing down the domain gap. Comprehensive experiments show that the proposed method can not only provide effective and intuitive explanations but also outperform previous state-of-the-arts. Code is available at \u0000<uri>https://drive.google.com/file/d/1b1EHFghiF1ExD-Cn1HYg75VutfkXWp60/view?usp=sharing</uri>\u0000.","PeriodicalId":94032,"journal":{"name":"IEEE transactions on image processing : a publication of the IEEE Signal Processing Society","volume":"33 ","pages":"5284-5297"},"PeriodicalIF":0.0,"publicationDate":"2024-09-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142245413","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-09-18DOI: 10.1109/TIP.2024.3459800
Ke Su;Xingxing Zhang;Siyang Zhang;Jun Zhu;Bo Zhang
Recently, there exists an increased research interest in embodied artificial intelligence (EAI), which involves an agent learning to perform a specific task when dynamically interacting with the surrounding 3D environment. There into, a new challenge is that many unseen objects may appear due to the increased number of object categories in 3D scenes. It makes developing models with strong zero-shot generalization ability to new objects necessary. Existing work tries to achieve this goal by providing embodied agents with massive high-quality human annotations closely related to the task to be learned, while it is too costly in practice. Inspired by recent advances in pre-trained models in 2D visual tasks, we attempt to boost zero-shot generalization for embodied reasoning with vision-language pre-training that can encode common sense as general prior knowledge. To further improve its performance on a specific task, we rectify the pre-trained representation through masked scene graph modeling (MSGM) in a self-supervised manner, where the task-specific knowledge is learned from iterative message passing. Our method can improve a variety of representative embodied reasoning tasks by a large margin (e.g., over 5.0% w.r.t. answer accuracy on MP3D-EQA dataset that consists of many real-world scenes with a large number of new objects during testing), and achieve the new state-of-the-art performance.
{"title":"To Boost Zero-Shot Generalization for Embodied Reasoning With Vision-Language Pre-Training","authors":"Ke Su;Xingxing Zhang;Siyang Zhang;Jun Zhu;Bo Zhang","doi":"10.1109/TIP.2024.3459800","DOIUrl":"10.1109/TIP.2024.3459800","url":null,"abstract":"Recently, there exists an increased research interest in embodied artificial intelligence (EAI), which involves an agent learning to perform a specific task when dynamically interacting with the surrounding 3D environment. There into, a new challenge is that many unseen objects may appear due to the increased number of object categories in 3D scenes. It makes developing models with strong zero-shot generalization ability to new objects necessary. Existing work tries to achieve this goal by providing embodied agents with massive high-quality human annotations closely related to the task to be learned, while it is too costly in practice. Inspired by recent advances in pre-trained models in 2D visual tasks, we attempt to boost zero-shot generalization for embodied reasoning with vision-language pre-training that can encode common sense as general prior knowledge. To further improve its performance on a specific task, we rectify the pre-trained representation through masked scene graph modeling (MSGM) in a self-supervised manner, where the task-specific knowledge is learned from iterative message passing. Our method can improve a variety of representative embodied reasoning tasks by a large margin (e.g., over 5.0% w.r.t. answer accuracy on MP3D-EQA dataset that consists of many real-world scenes with a large number of new objects during testing), and achieve the new state-of-the-art performance.","PeriodicalId":94032,"journal":{"name":"IEEE transactions on image processing : a publication of the IEEE Signal Processing Society","volume":"33 ","pages":"5370-5381"},"PeriodicalIF":0.0,"publicationDate":"2024-09-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142245420","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-09-18DOI: 10.1109/TIP.2024.3457246
Long Chen;Yunzhou Xie;Yaxin Li;Qi Xu;Junyu Dong
Autonomous underwater vehicles (AUVs) equipped with the intelligent underwater object detection technique is of great significance for underwater navigation. Advanced underwater object detection frameworks adopt skip connections to enhance the feature representation which further boosts the detection precision. However, we reveal two limitations of standard skip connections: 1) standard skip connections do not consider the feature heterogeneity, resulting in a sub-optimal feature fusion strategy; 2) feature redundancy exists in the skip connected features that not all the channels in the fused feature maps are equally important, the network learning should focus on the informative channels rather than the redundant ones. In this paper, we propose a novel channel-weighted skip connection network (CWSCNet) to learn multiple hyper fusion features for improving multi-scale underwater object detection. In CWSCNet, a novel feature fusion module, named channel-weighted skip connection (CWSC), is proposed to adaptively adjust the importance of different channels during feature fusion. The CWSC module removes feature heterogeneity that strengthens the compatibility of different feature maps, it also works as an effective feature selection strategy that enables CWSCNet to focus on learning channels with more object-related information. Extensive experiments on three underwater object detection datasets RUOD, URPC2017 and URPC2018 show that the proposed CWSCNet achieves comparable or state-of-the-art performances in underwater object detection.
{"title":"CWSCNet: Channel-Weighted Skip Connection Network for Underwater Object Detection","authors":"Long Chen;Yunzhou Xie;Yaxin Li;Qi Xu;Junyu Dong","doi":"10.1109/TIP.2024.3457246","DOIUrl":"10.1109/TIP.2024.3457246","url":null,"abstract":"Autonomous underwater vehicles (AUVs) equipped with the intelligent underwater object detection technique is of great significance for underwater navigation. Advanced underwater object detection frameworks adopt skip connections to enhance the feature representation which further boosts the detection precision. However, we reveal two limitations of standard skip connections: 1) standard skip connections do not consider the feature heterogeneity, resulting in a sub-optimal feature fusion strategy; 2) feature redundancy exists in the skip connected features that not all the channels in the fused feature maps are equally important, the network learning should focus on the informative channels rather than the redundant ones. In this paper, we propose a novel channel-weighted skip connection network (CWSCNet) to learn multiple hyper fusion features for improving multi-scale underwater object detection. In CWSCNet, a novel feature fusion module, named channel-weighted skip connection (CWSC), is proposed to adaptively adjust the importance of different channels during feature fusion. The CWSC module removes feature heterogeneity that strengthens the compatibility of different feature maps, it also works as an effective feature selection strategy that enables CWSCNet to focus on learning channels with more object-related information. Extensive experiments on three underwater object detection datasets RUOD, URPC2017 and URPC2018 show that the proposed CWSCNet achieves comparable or state-of-the-art performances in underwater object detection.","PeriodicalId":94032,"journal":{"name":"IEEE transactions on image processing : a publication of the IEEE Signal Processing Society","volume":"33 ","pages":"5206-5218"},"PeriodicalIF":0.0,"publicationDate":"2024-09-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142245467","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-09-17DOI: 10.1109/TIP.2024.3458858
Xi Yang;Dechen Kong;Ren Lin;Nannan Wang;Xinbo Gao
Most few-shot learning methods employ either adaptive approaches or parameter amortization techniques. However, their reliance on pre-trained models presents a significant vulnerability. When an attacker’s trigger activates a hidden backdoor, it may result in the misclassification of images, profoundly affecting the model’s performance. In our research, we explore adaptive defenses against backdoor attacks for few-shot learning. We introduce a specialized stochastic process tailored to task characteristics that safeguards the classification model against attack-induced incorrect feature extraction. This process functions during forward propagation and is thus termed an “in-process defense.” Our method employs an adaptive strategy, effectively generating task-level representations, enabling rapid adaptation to pre-trained models, and proving effective in few-shot classification scenarios for countering backdoor attacks. We apply latent stochastic processes to approximate task distributions and derive task-level representations from the support set. This task-level representation guides feature extraction, leading to backdoor trigger mismatching and forming the foundation of our parameter defense strategy. Benchmark tests on Meta-Dataset reveal that our approach not only withstands backdoor attacks but also shows an improved adaptation in addressing few-shot classification tasks.
{"title":"Adapting Few-Shot Classification via In-Process Defense","authors":"Xi Yang;Dechen Kong;Ren Lin;Nannan Wang;Xinbo Gao","doi":"10.1109/TIP.2024.3458858","DOIUrl":"10.1109/TIP.2024.3458858","url":null,"abstract":"Most few-shot learning methods employ either adaptive approaches or parameter amortization techniques. However, their reliance on pre-trained models presents a significant vulnerability. When an attacker’s trigger activates a hidden backdoor, it may result in the misclassification of images, profoundly affecting the model’s performance. In our research, we explore adaptive defenses against backdoor attacks for few-shot learning. We introduce a specialized stochastic process tailored to task characteristics that safeguards the classification model against attack-induced incorrect feature extraction. This process functions during forward propagation and is thus termed an “in-process defense.” Our method employs an adaptive strategy, effectively generating task-level representations, enabling rapid adaptation to pre-trained models, and proving effective in few-shot classification scenarios for countering backdoor attacks. We apply latent stochastic processes to approximate task distributions and derive task-level representations from the support set. This task-level representation guides feature extraction, leading to backdoor trigger mismatching and forming the foundation of our parameter defense strategy. Benchmark tests on Meta-Dataset reveal that our approach not only withstands backdoor attacks but also shows an improved adaptation in addressing few-shot classification tasks.","PeriodicalId":94032,"journal":{"name":"IEEE transactions on image processing : a publication of the IEEE Signal Processing Society","volume":"33 ","pages":"5232-5245"},"PeriodicalIF":0.0,"publicationDate":"2024-09-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142236317","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-09-17DOI: 10.1109/TIP.2024.3458854
Bo Dang;Yansheng Li;Yongjun Zhang;Jiayi Ma
Semi-supervised semantic segmentation focuses on the exploration of a small amount of labeled data and a large amount of unlabeled data, which is more in line with the demands of real-world image understanding applications. However, it is still hindered by the inability to fully and effectively leverage unlabeled images. In this paper, we reveal that cross-window consistency (CWC) is helpful in comprehensively extracting auxiliary supervision from unlabeled data. Additionally, we propose a novel CWC-driven progressive learning framework to optimize the deep network by mining weak-to-strong constraints from massive unlabeled data. More specifically, this paper presents a biased cross-window consistency (BCC) loss with an importance factor, which helps the deep network explicitly constrain confidence maps from overlapping regions in different windows to maintain semantic consistency with larger contexts. In addition, we propose a dynamic pseudo-label memory bank (DPM) to provide high-consistency and high-reliability pseudo-labels to further optimize the network. Extensive experiments on three representative datasets of urban views, medical scenarios, and satellite scenes with consistent performance gain demonstrate the superiority of our framework. Our code is released at https://jack-bo1220.github.io/project/CWC.html .
{"title":"Progressive Learning With Cross-Window Consistency for Semi-Supervised Semantic Segmentation","authors":"Bo Dang;Yansheng Li;Yongjun Zhang;Jiayi Ma","doi":"10.1109/TIP.2024.3458854","DOIUrl":"10.1109/TIP.2024.3458854","url":null,"abstract":"Semi-supervised semantic segmentation focuses on the exploration of a small amount of labeled data and a large amount of unlabeled data, which is more in line with the demands of real-world image understanding applications. However, it is still hindered by the inability to fully and effectively leverage unlabeled images. In this paper, we reveal that cross-window consistency (CWC) is helpful in comprehensively extracting auxiliary supervision from unlabeled data. Additionally, we propose a novel CWC-driven progressive learning framework to optimize the deep network by mining weak-to-strong constraints from massive unlabeled data. More specifically, this paper presents a biased cross-window consistency (BCC) loss with an importance factor, which helps the deep network explicitly constrain confidence maps from overlapping regions in different windows to maintain semantic consistency with larger contexts. In addition, we propose a dynamic pseudo-label memory bank (DPM) to provide high-consistency and high-reliability pseudo-labels to further optimize the network. Extensive experiments on three representative datasets of urban views, medical scenarios, and satellite scenes with consistent performance gain demonstrate the superiority of our framework. Our code is released at \u0000<uri>https://jack-bo1220.github.io/project/CWC.html</uri>\u0000.","PeriodicalId":94032,"journal":{"name":"IEEE transactions on image processing : a publication of the IEEE Signal Processing Society","volume":"33 ","pages":"5219-5231"},"PeriodicalIF":0.0,"publicationDate":"2024-09-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142236613","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：