Pattern Recognition最新文献_第3页

Improving stability and performance of spiking neural networks through enhancing temporal consistency 通过增强时间一致性提高尖峰神经网络的稳定性和性能

IF 7.5 1区计算机科学 Q1 COMPUTER SCIENCE, ARTIFICIAL INTELLIGENCE

Pattern Recognition

Pub Date : 2024-10-26 DOI: 10.1016/j.patcog.2024.111094

Spiking neural networks have gained significant attention due to their brain-like information processing capabilities. The use of surrogate gradients has made it possible to train spiking neural networks with backpropagation, leading to impressive performance in various tasks. However, spiking neural networks trained with backpropagation typically approximate actual labels using the average output, often necessitating a larger simulation timestep to enhance the network’s performance. This delay constraint poses a challenge to the further advancement of spiking neural networks. Current training algorithms tend to overlook the differences in output distribution at various timesteps. Particularly for neuromorphic datasets, inputs at different timesteps can cause inconsistencies in output distribution, leading to a significant deviation from the optimal direction when combining optimization directions from different moments. To tackle this issue, we have designed a method to enhance the temporal consistency of outputs at different timesteps. We have conducted experiments on static datasets such as CIFAR10, CIFAR100, and ImageNet. The results demonstrate that our algorithm can achieve comparable performance to other optimal SNN algorithms. Notably, our algorithm has achieved state-of-the-art performance on neuromorphic datasets DVS-CIFAR10 and N-Caltech101, and can achieve superior performance in the test phase with timestep

T

= 1.

尖峰神经网络因其类似大脑的信息处理能力而备受关注。代梯度的使用使得利用反向传播训练尖峰神经网络成为可能，从而在各种任务中取得了令人印象深刻的性能。然而，使用反向传播训练的尖峰神经网络通常使用平均输出来逼近实际标签，因此往往需要更大的模拟时间步来提高网络性能。这种延迟限制为尖峰神经网络的进一步发展带来了挑战。当前的训练算法往往会忽略不同时间步的输出分布差异。特别是对于神经形态数据集而言，不同时间步的输入会导致输出分布的不一致性，从而导致在结合不同时刻的优化方向时与最优方向产生显著偏差。为了解决这个问题，我们设计了一种方法来增强不同时间步输出的时间一致性。我们在 CIFAR10、CIFAR100 和 ImageNet 等静态数据集上进行了实验。结果表明，我们的算法可以达到与其他最优 SNN 算法相当的性能。值得注意的是，我们的算法在神经形态数据集 DVS-CIFAR10 和 N-Caltech101 上取得了最先进的性能，并能在时间步 T = 1 的测试阶段取得优异的性能。

{"title":"Improving stability and performance of spiking neural networks through enhancing temporal consistency","authors":"","doi":"10.1016/j.patcog.2024.111094","DOIUrl":"10.1016/j.patcog.2024.111094","url":null,"abstract":"<div><div>Spiking neural networks have gained significant attention due to their brain-like information processing capabilities. The use of surrogate gradients has made it possible to train spiking neural networks with backpropagation, leading to impressive performance in various tasks. However, spiking neural networks trained with backpropagation typically approximate actual labels using the average output, often necessitating a larger simulation timestep to enhance the network’s performance. This delay constraint poses a challenge to the further advancement of spiking neural networks. Current training algorithms tend to overlook the differences in output distribution at various timesteps. Particularly for neuromorphic datasets, inputs at different timesteps can cause inconsistencies in output distribution, leading to a significant deviation from the optimal direction when combining optimization directions from different moments. To tackle this issue, we have designed a method to enhance the temporal consistency of outputs at different timesteps. We have conducted experiments on static datasets such as CIFAR10, CIFAR100, and ImageNet. The results demonstrate that our algorithm can achieve comparable performance to other optimal SNN algorithms. Notably, our algorithm has achieved state-of-the-art performance on neuromorphic datasets DVS-CIFAR10 and N-Caltech101, and can achieve superior performance in the test phase with timestep <span><math><mi>T</mi></math></span> = 1.</div></div>","PeriodicalId":49713,"journal":{"name":"Pattern Recognition","volume":null,"pages":null},"PeriodicalIF":7.5,"publicationDate":"2024-10-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142572364","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Continual learning with high-order experience replay for dynamic network embedding 利用高阶经验回放进行持续学习，实现动态网络嵌入

IF 7.5 1区计算机科学 Q1 COMPUTER SCIENCE, ARTIFICIAL INTELLIGENCE

Pattern Recognition

Pub Date : 2024-10-26 DOI: 10.1016/j.patcog.2024.111093

Dynamic network embedding (DNE) poses a tough challenge in graph representation learning, especially when confronted with the frequent updates of streaming data. Conventional DNEs primarily resort to parameter updating but perform inadequately on historical networks, resulting in the problem of catastrophic forgetting. To tackle such issues, recent advancements in graph neural networks (GNNs) have explored matrix factorization techniques. However, these approaches encounter difficulties in preserving the global patterns of incremental data. In this paper, we propose CLDNE, a Continual Learning framework specifically designed for Dynamic Network Embedding. At the core of CLDNE lies a streaming graph auto-encoder that effectively captures both global and local patterns of the input graph. To further overcome catastrophic forgetting, CLDNE is equipped with an experience replay buffer and a knowledge distillation module, which preserve high-order historical topology and static historical patterns. We conduct experiments on four dynamic networks using link prediction and node classification tasks to evaluate the effectiveness of CLDNE. The outcomes demonstrate that CLDNE successfully mitigates the catastrophic forgetting problem and reduces training time by 80% without a significant loss in learning new patterns.

动态网络嵌入（DNE）对图表示学习提出了严峻的挑战，尤其是在面对频繁更新的流数据时。传统的 DNE 主要依靠参数更新，但在历史网络上表现不佳，导致灾难性遗忘问题。为了解决这些问题，图神经网络（GNN）的最新进展是探索矩阵因式分解技术。然而，这些方法在保留增量数据的全局模式方面遇到了困难。在本文中，我们提出了专门为动态网络嵌入设计的持续学习框架 CLDNE。CLDNE 的核心是流图自动编码器，它能有效捕捉输入图的全局和局部模式。为了进一步克服灾难性遗忘，CLDNE 配备了经验重放缓冲区和知识提炼模块，以保存高阶历史拓扑和静态历史模式。我们使用链接预测和节点分类任务在四个动态网络上进行了实验，以评估 CLDNE 的有效性。实验结果表明，CLDNE 成功地缓解了灾难性遗忘问题，并将训练时间缩短了 80%，而且在学习新模式方面没有明显损失。

{"title":"Continual learning with high-order experience replay for dynamic network embedding","authors":"","doi":"10.1016/j.patcog.2024.111093","DOIUrl":"10.1016/j.patcog.2024.111093","url":null,"abstract":"<div><div>Dynamic network embedding (DNE) poses a tough challenge in graph representation learning, especially when confronted with the frequent updates of streaming data. Conventional DNEs primarily resort to parameter updating but perform inadequately on historical networks, resulting in the problem of catastrophic forgetting. To tackle such issues, recent advancements in graph neural networks (GNNs) have explored matrix factorization techniques. However, these approaches encounter difficulties in preserving the global patterns of incremental data. In this paper, we propose CLDNE, a <u>C</u>ontinual <u>L</u>earning framework specifically designed for <u>D</u>ynamic <u>N</u>etwork <u>E</u>mbedding. At the core of CLDNE lies a streaming graph auto-encoder that effectively captures both global and local patterns of the input graph. To further overcome catastrophic forgetting, CLDNE is equipped with an experience replay buffer and a knowledge distillation module, which preserve high-order historical topology and static historical patterns. We conduct experiments on four dynamic networks using link prediction and node classification tasks to evaluate the effectiveness of CLDNE. The outcomes demonstrate that CLDNE successfully mitigates the catastrophic forgetting problem and reduces training time by 80% without a significant loss in learning new patterns.</div></div>","PeriodicalId":49713,"journal":{"name":"Pattern Recognition","volume":null,"pages":null},"PeriodicalIF":7.5,"publicationDate":"2024-10-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142594030","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

ECTFormer: An efficient Conv-Transformer model design for image recognition ECTFormer：用于图像识别的高效 Conv-Transformer 模型设计

IF 7.5 1区计算机科学 Q1 COMPUTER SCIENCE, ARTIFICIAL INTELLIGENCE

Pattern Recognition

Pub Date : 2024-10-25 DOI: 10.1016/j.patcog.2024.111092

Since the success of Vision Transformers (ViTs), there has been growing interest in combining ConvNets and Transformers in the computer vision community. While the hybrid models have demonstrated state-of-the-art performance, many of these models are too large and complex to be applied to edge devices for real-world applications. To address this challenge, we propose an efficient hybrid network called ECTFormer that leverages the strengths of ConvNets and Transformers while considering both model performance and inference speed. Specifically, our approach involves: (1) optimizing the combination of convolution kernels by dynamically adjusting kernel sizes based on the scale of feature tensors; (2) revisiting existing overlapping patchify to not only reduce the model size but also propagate fine-grained patches for the performance enhancement; and (3) introducing an efficient single-head self-attention mechanism, rather than multi-head self-attention in the base Transformer, to minimize the increase in model size and boost inference speed, overcoming bottlenecks of ViTs. In experimental results on ImageNet-1K, ECTFormer not only demonstrates comparable or higher top-1 accuracy but also faster inference speed on both GPUs and edge devices compared to other efficient networks.

自视觉变换器（ViTs）取得成功以来，计算机视觉界对 ConvNets 和变换器的结合越来越感兴趣。虽然混合模型已经展示了最先进的性能，但其中许多模型过于庞大和复杂，无法应用于现实世界中的边缘设备。为了应对这一挑战，我们提出了一种名为 ECTFormer 的高效混合网络，它充分利用了 ConvNets 和 Transformers 的优势，同时兼顾了模型性能和推理速度。具体来说，我们的方法包括：(1) 根据特征张量的尺度动态调整内核大小，从而优化卷积内核的组合；(2) 重新审视现有的重叠补丁，不仅减小模型大小，而且传播细粒度补丁以提高性能；(3) 引入高效的单头自关注机制，而不是基础变换器中的多头自关注机制，从而最大限度地减小模型大小的增加，提高推理速度，克服 ViTs 的瓶颈。在 ImageNet-1K 的实验结果中，与其他高效网络相比，ECTFormer 不仅在 GPU 和边缘设备上表现出相当或更高的 top-1 精度，而且推理速度也更快。

{"title":"ECTFormer: An efficient Conv-Transformer model design for image recognition","authors":"","doi":"10.1016/j.patcog.2024.111092","DOIUrl":"10.1016/j.patcog.2024.111092","url":null,"abstract":"<div><div>Since the success of Vision Transformers (ViTs), there has been growing interest in combining ConvNets and Transformers in the computer vision community. While the hybrid models have demonstrated state-of-the-art performance, many of these models are too large and complex to be applied to edge devices for real-world applications. To address this challenge, we propose an efficient hybrid network called ECTFormer that leverages the strengths of ConvNets and Transformers while considering both model performance and inference speed. Specifically, our approach involves: (1) optimizing the combination of convolution kernels by dynamically adjusting kernel sizes based on the scale of feature tensors; (2) revisiting existing overlapping patchify to not only reduce the model size but also propagate fine-grained patches for the performance enhancement; and (3) introducing an efficient single-head self-attention mechanism, rather than multi-head self-attention in the base Transformer, to minimize the increase in model size and boost inference speed, overcoming bottlenecks of ViTs. In experimental results on ImageNet-1K, ECTFormer not only demonstrates comparable or higher top-1 accuracy but also faster inference speed on both GPUs and edge devices compared to other efficient networks.</div></div>","PeriodicalId":49713,"journal":{"name":"Pattern Recognition","volume":null,"pages":null},"PeriodicalIF":7.5,"publicationDate":"2024-10-25","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142553050","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Multi-view visual semantic embedding for cross-modal image–text retrieval 用于跨模态图像文本检索的多视图视觉语义嵌入

IF 7.5 1区计算机科学 Q1 COMPUTER SCIENCE, ARTIFICIAL INTELLIGENCE

Pattern Recognition

Pub Date : 2024-10-24 DOI: 10.1016/j.patcog.2024.111088

Visual Semantic Embedding (VSE) is a dominant method for cross-modal image–text retrieval. The purpose of VSE is to learn an embedding space where images can be embedded close to the corresponding captions. However, there are large intra-class variations in image–text data. Multiple captions describing the same image may be described from different views, and descriptions of different views are often dissimilar. The VSE method embeds samples from the same class in similar positions, which suppresses intra-class variations and leads to inferior generalization. This paper proposes a Multi-View Visual Semantic Embedding (MV-VSE) framework that learns multiple embeddings for an image, explicitly modeling intra-class variation. To optimize the MV-VSE framework, a multi-view triplet loss is proposed, which jointly optimizes multi-view embeddings while retaining intra-class variation. Recently, large-scale Vision-Language Pre-training (VLP) has become a new paradigm for cross-modal image–text retrieval. To allow our framework to be flexibly applied to the traditional VSE models and VSE-based VLP models, we incorporate the contrastive loss commonly used in VLP and the triplet loss into a unified loss, and further propose a multi-view unified loss. Our framework can be applied plug-and-play to traditional VSE models and VSE-based VLP models without excessively increasing model complexity. Experimental results on the image–text retrieval benchmark datasets demonstrate that applying our framework can boost the retrieval performance of current VSE models. The code is available at https://github.com/AAA-Zheng/MV-VSE.

视觉语义嵌入（VSE）是跨模态图像-文本检索的主要方法。VSE 的目的是学习一个嵌入空间，在这个空间中，图像可以嵌入到相应的标题附近。然而，图像-文本数据存在很大的类内差异。描述同一图像的多个标题可能来自不同的视图，而不同视图的描述往往是不同的。VSE 方法将同类样本嵌入相似位置，抑制了类内差异，导致泛化效果较差。本文提出了一种多视图视觉语义嵌入（MV-VSE）框架，它可以学习图像的多个嵌入，明确地模拟类内变化。为了优化 MV-VSE 框架，本文提出了一种多视图三重损失（multi-view triplet loss），它可以在保留类内变化的同时联合优化多视图嵌入。最近，大规模视觉语言预训练（VLP）已成为跨模态图像-文本检索的新范例。为了使我们的框架能够灵活地应用于传统的 VSE 模型和基于 VSE 的 VLP 模型，我们将 VLP 中常用的对比度损失和三重损失合并为统一损失，并进一步提出了多视图统一损失。我们的框架可以即插即用地应用于传统的 VSE 模型和基于 VSE 的 VLP 模型，而不会过度增加模型的复杂性。在图像-文本检索基准数据集上的实验结果表明，应用我们的框架可以提高当前 VSE 模型的检索性能。代码可在 https://github.com/AAA-Zheng/MV-VSE 上获取。

{"title":"Multi-view visual semantic embedding for cross-modal image–text retrieval","authors":"","doi":"10.1016/j.patcog.2024.111088","DOIUrl":"10.1016/j.patcog.2024.111088","url":null,"abstract":"<div><div>Visual Semantic Embedding (VSE) is a dominant method for cross-modal image–text retrieval. The purpose of VSE is to learn an embedding space where images can be embedded close to the corresponding captions. However, there are large intra-class variations in image–text data. Multiple captions describing the same image may be described from different views, and descriptions of different views are often dissimilar. The VSE method embeds samples from the same class in similar positions, which suppresses intra-class variations and leads to inferior generalization. This paper proposes a Multi-View Visual Semantic Embedding (MV-VSE) framework that learns multiple embeddings for an image, explicitly modeling intra-class variation. To optimize the MV-VSE framework, a multi-view triplet loss is proposed, which jointly optimizes multi-view embeddings while retaining intra-class variation. Recently, large-scale Vision-Language Pre-training (VLP) has become a new paradigm for cross-modal image–text retrieval. To allow our framework to be flexibly applied to the traditional VSE models and VSE-based VLP models, we incorporate the contrastive loss commonly used in VLP and the triplet loss into a unified loss, and further propose a multi-view unified loss. Our framework can be applied plug-and-play to traditional VSE models and VSE-based VLP models without excessively increasing model complexity. Experimental results on the image–text retrieval benchmark datasets demonstrate that applying our framework can boost the retrieval performance of current VSE models. The code is available at <span><span>https://github.com/AAA-Zheng/MV-VSE</span><svg><path></path></svg></span>.</div></div>","PeriodicalId":49713,"journal":{"name":"Pattern Recognition","volume":null,"pages":null},"PeriodicalIF":7.5,"publicationDate":"2024-10-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142572365","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Weakly Supervised Underwater Object Real-time Detection Based on High-resolution Attention Class Activation Mapping and Category Hierarchy 基于高分辨率注意力类别激活映射和类别层次结构的弱监督水下物体实时检测

IF 7.5 1区计算机科学 Q1 COMPUTER SCIENCE, ARTIFICIAL INTELLIGENCE

Pattern Recognition

Pub Date : 2024-10-24 DOI: 10.1016/j.patcog.2024.111111

Recently, deep learning-based underwater object detection technology has achieved remarkable success. However, the accuracy and completeness of dataset instance annotation are crucial for its success. The quality of underwater images is low, severe objects clustering, and occlusion, acquiring object's annotations demands substantial time and labor costs, while mis annotation and missed annotation can also degrade model performance and limit their application in practical scenarios. To address this issue, this paper presents a novel weakly supervised underwater object real-time detection method, which is divided into two subtasks: weakly supervised object localization and real-time object detection. In the weakly supervised object localization task, we design a novel category hierarchy structure network that integrates the high-resolution attention-class activation mapping algorithm to obtain high-quality object class activation maps, weaken background interference, and obtain more complete object regions. The parameterized spatial loss module is devised to enable the model to escape from local optimal solutions, thus accurately and efficiently obtaining object pseudo-detection annotation boxes. For the real-time object detection task, the single-stage detector YOLOv7 is selected as the basic detection model, and an object perception loss function is designed based on the class activation map to jointly supervise the training process. A method for filtering noisy pseudo-supervision information is proposed to enhance the pseudo-supervision information involved in training. Ablation experiments and multi-method comparison experiments were conducted on the URPC and RUOD datasets, and the results verify the effectiveness of the proposed strategy, and our model exhibits significant advantages in detection performance and detection efficiency compared to current mainstream and advanced models.

最近，基于深度学习的水下物体检测技术取得了显著成就。然而，数据集实例标注的准确性和完整性是其成功的关键。水下图像质量低、物体聚类和遮挡现象严重，获取物体标注需要大量的时间和人力成本，而错误标注和遗漏标注也会降低模型性能，限制其在实际场景中的应用。针对这一问题，本文提出了一种新型的弱监督水下物体实时检测方法，该方法分为两个子任务：弱监督物体定位和物体实时检测。在弱监督物体定位任务中，我们设计了一种新颖的类别分层结构网络，该网络集成了高分辨率注意力-类别激活映射算法，可获得高质量的物体类别激活映射图，削弱背景干扰，获得更完整的物体区域。参数化的空间损失模块使模型能够摆脱局部最优解，从而准确高效地获得物体伪检测注释框。针对实时物体检测任务，选择单级检测器 YOLOv7 作为基本检测模型，并设计了基于类激活图的物体感知损失函数来共同监督训练过程。提出了一种过滤噪声伪监督信息的方法，以增强训练中涉及的伪监督信息。在 URPC 和 RUOD 数据集上进行了消融实验和多方法对比实验，结果验证了所提策略的有效性，与目前的主流模型和先进模型相比，我们的模型在检测性能和检测效率上具有显著优势。

{"title":"Weakly Supervised Underwater Object Real-time Detection Based on High-resolution Attention Class Activation Mapping and Category Hierarchy","authors":"","doi":"10.1016/j.patcog.2024.111111","DOIUrl":"10.1016/j.patcog.2024.111111","url":null,"abstract":"<div><div>Recently, deep learning-based underwater object detection technology has achieved remarkable success. However, the accuracy and completeness of dataset instance annotation are crucial for its success. The quality of underwater images is low, severe objects clustering, and occlusion, acquiring object's annotations demands substantial time and labor costs, while mis annotation and missed annotation can also degrade model performance and limit their application in practical scenarios. To address this issue, this paper presents a novel weakly supervised underwater object real-time detection method, which is divided into two subtasks: weakly supervised object localization and real-time object detection. In the weakly supervised object localization task, we design a novel category hierarchy structure network that integrates the high-resolution attention-class activation mapping algorithm to obtain high-quality object class activation maps, weaken background interference, and obtain more complete object regions. The parameterized spatial loss module is devised to enable the model to escape from local optimal solutions, thus accurately and efficiently obtaining object pseudo-detection annotation boxes. For the real-time object detection task, the single-stage detector YOLOv7 is selected as the basic detection model, and an object perception loss function is designed based on the class activation map to jointly supervise the training process. A method for filtering noisy pseudo-supervision information is proposed to enhance the pseudo-supervision information involved in training. Ablation experiments and multi-method comparison experiments were conducted on the URPC and RUOD datasets, and the results verify the effectiveness of the proposed strategy, and our model exhibits significant advantages in detection performance and detection efficiency compared to current mainstream and advanced models.</div></div>","PeriodicalId":49713,"journal":{"name":"Pattern Recognition","volume":null,"pages":null},"PeriodicalIF":7.5,"publicationDate":"2024-10-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142572366","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Global-aware Fragment Representation Aggregation Network for image–text retrieval 用于图像文本检索的全局感知片段表示聚合网络

IF 7.5 1区计算机科学 Q1 COMPUTER SCIENCE, ARTIFICIAL INTELLIGENCE

Pattern Recognition

Pub Date : 2024-10-24 DOI: 10.1016/j.patcog.2024.111085

Image–text retrieval is an important kind of cross-modal retrieval method and has recently attracted much attention. Existing image–text retrieval methods often ignore the relative importance of each fragment (region in an image or word in a sentence) on the global semantic of image or text when aggregating features of image or text fragments, resulting in the ineffectiveness of the learned image and text representations. To address this problem, we propose an image–text retrieval method named Global-aware Fragment Representation Aggregation Network (GFRAN). Specifically, it first designs a fine-grained multimodal information interaction module based on the self-attention mechanism to model both the intra-modality and inter-modality relationships between image regions and words. Then, with the guidance of the global image or text feature, it aggregates image or text fragment features conditioned on their attention weights over global feature, to highlight fragments that contribute more to the overall semantics of images and texts. Extensive experiments on two benchmark datasets Flickr30K and MS-COCO demonstrate the superiority of the proposed GFRAN model over several state-of-the-art baselines.

图像-文本检索是一种重要的跨模态检索方法，近年来备受关注。现有的图像-文本检索方法在聚合图像或文本片段特征时，往往忽略了每个片段（图像中的区域或句子中的单词）对图像或文本全局语义的相对重要性，导致学习到的图像和文本表征效果不佳。针对这一问题，我们提出了一种名为全局感知片段表征聚合网络（GFRAN）的图像文本检索方法。具体来说，它首先设计了一个基于自我注意机制的细粒度多模态信息交互模块，对图像区域和文字之间的模态内和模态间关系进行建模。然后，在全局图像或文本特征的指导下，根据图像或文本片段在全局特征上的注意力权重，聚合图像或文本片段特征，以突出对图像和文本整体语义贡献更大的片段。在 Flickr30K 和 MS-COCO 这两个基准数据集上进行的大量实验证明，所提出的 GFRAN 模型优于几种最先进的基线模型。

{"title":"Global-aware Fragment Representation Aggregation Network for image–text retrieval","authors":"","doi":"10.1016/j.patcog.2024.111085","DOIUrl":"10.1016/j.patcog.2024.111085","url":null,"abstract":"<div><div>Image–text retrieval is an important kind of cross-modal retrieval method and has recently attracted much attention. Existing image–text retrieval methods often ignore the relative importance of each fragment (region in an image or word in a sentence) on the global semantic of image or text when aggregating features of image or text fragments, resulting in the ineffectiveness of the learned image and text representations. To address this problem, we propose an image–text retrieval method named Global-aware Fragment Representation Aggregation Network (GFRAN). Specifically, it first designs a fine-grained multimodal information interaction module based on the self-attention mechanism to model both the intra-modality and inter-modality relationships between image regions and words. Then, with the guidance of the global image or text feature, it aggregates image or text fragment features conditioned on their attention weights over global feature, to highlight fragments that contribute more to the overall semantics of images and texts. Extensive experiments on two benchmark datasets Flickr30K and MS-COCO demonstrate the superiority of the proposed GFRAN model over several state-of-the-art baselines.</div></div>","PeriodicalId":49713,"journal":{"name":"Pattern Recognition","volume":null,"pages":null},"PeriodicalIF":7.5,"publicationDate":"2024-10-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142572370","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Approximate geometric structure transfer for cross-domain image classification 用于跨域图像分类的近似几何结构转移

IF 7.5 1区计算机科学 Q1 COMPUTER SCIENCE, ARTIFICIAL INTELLIGENCE

Pattern Recognition

Pub Date : 2024-10-24 DOI: 10.1016/j.patcog.2024.111105

The main purpose of domain adaptation (DA) is to conduct cross-domain related knowledge transfer. Considering the issue of unsupervised DA (UDA), learning a transformation that reduces the differences between domains is the primary goal. In addition to minimizing both the marginal and conditional distributions between the source and target domains, many methods explore potential factors that show the commonalities among domains to yield improved learning efficiency. However, geometric structure information is overlooked by most existing approaches, indicating that the shared information between domains has not been fully exploited. On account of this finding, by taking advantage of more potential shared factors to further enhance the results of DA, we propose an approximate geometric structure transfer (AGST) method for cross-domain image classification in this paper. By combining structural consistency and sample reweighting techniques, AGST encodes the geometric structure information taken from the samples in both domains, enabling it to easily obtain richer interdomain features and effectively facilitate knowledge transfer. Extensive experiments are conducted on several cross-domain data benchmarks. The experimental results indicate that our AGST method can outperform many state-of-the-art algorithms.

领域适应（DA）的主要目的是进行跨领域相关知识转移。考虑到无监督领域适应（UDA）问题，学习一种能减少领域间差异的转换是首要目标。除了最小化源域和目标域之间的边际分布和条件分布外，许多方法还探索显示域间共性的潜在因素，以提高学习效率。然而，大多数现有方法都忽略了几何结构信息，这表明域之间的共享信息尚未得到充分利用。基于这一发现，通过利用更多潜在的共享因素来进一步提高 DA 的结果，我们在本文中提出了一种用于跨域图像分类的近似几何结构转移（AGST）方法。通过结合结构一致性和样本重权技术，AGST 编码了从两个域的样本中提取的几何结构信息，使其能够轻松获得更丰富的域间特征，并有效促进知识转移。我们在多个跨域数据基准上进行了广泛的实验。实验结果表明，我们的 AGST 方法优于许多最先进的算法。

{"title":"Approximate geometric structure transfer for cross-domain image classification","authors":"","doi":"10.1016/j.patcog.2024.111105","DOIUrl":"10.1016/j.patcog.2024.111105","url":null,"abstract":"<div><div>The main purpose of domain adaptation (DA) is to conduct cross-domain related knowledge transfer. Considering the issue of unsupervised DA (UDA), learning a transformation that reduces the differences between domains is the primary goal. In addition to minimizing both the marginal and conditional distributions between the source and target domains, many methods explore potential factors that show the commonalities among domains to yield improved learning efficiency. However, geometric structure information is overlooked by most existing approaches, indicating that the shared information between domains has not been fully exploited. On account of this finding, by taking advantage of more potential shared factors to further enhance the results of DA, we propose an approximate geometric structure transfer (AGST) method for cross-domain image classification in this paper. By combining structural consistency and sample reweighting techniques, AGST encodes the geometric structure information taken from the samples in both domains, enabling it to easily obtain richer interdomain features and effectively facilitate knowledge transfer. Extensive experiments are conducted on several cross-domain data benchmarks. The experimental results indicate that our AGST method can outperform many state-of-the-art algorithms.</div></div>","PeriodicalId":49713,"journal":{"name":"Pattern Recognition","volume":null,"pages":null},"PeriodicalIF":7.5,"publicationDate":"2024-10-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142572369","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

NAS-BNN: Neural Architecture Search for Binary Neural Networks NAS-BNN：二元神经网络的神经架构搜索

IF 7.5 1区计算机科学 Q1 COMPUTER SCIENCE, ARTIFICIAL INTELLIGENCE

Pattern Recognition

Pub Date : 2024-10-23 DOI: 10.1016/j.patcog.2024.111086

Binary Neural Networks (BNNs) have gained extensive attention for their superior inferencing efficiency and compression ratio compared to traditional full-precision networks. However, due to the unique characteristics of BNNs, designing a powerful binary architecture is challenging and often requires significant manpower. A promising solution is to utilize Neural Architecture Search (NAS) to assist in designing BNNs, but current NAS methods for BNNs are relatively straightforward and leave a performance gap between the searched models and manually designed ones. To address this gap, we propose a novel neural architecture search scheme for binary neural networks, named NAS-BNN. We first carefully design a search space based on the unique characteristics of BNNs. Then, we present three training strategies, which significantly enhance the training of supernet and boost the performance of all subnets. Our discovered binary model family outperforms previous BNNs for a wide range of operations (OPs) from 20M to 200M. For instance, we achieve 68.20% top-1 accuracy on ImageNet with only 57M OPs. In addition, we validate the transferability of these searched BNNs on the object detection task, and our binary detectors with the searched BNNs achieve a novel state-of-the-art result, e.g., 31.6% mAP with 370M OPs, on MS COCO dataset. The source code and models will be released at https://github.com/VDIGPKU/NAS-BNN.

二元神经网络（BNN）因其优于传统全精度网络的推理效率和压缩比而受到广泛关注。然而，由于二元神经网络的独特性，设计一个强大的二元架构极具挑战性，通常需要大量人力。一个有前途的解决方案是利用神经架构搜索（NAS）来辅助设计 BNN，但目前针对 BNN 的 NAS 方法相对简单，搜索到的模型与人工设计的模型之间存在性能差距。针对这一差距，我们提出了一种新颖的二元神经网络神经架构搜索方案，命名为 NAS-BNN。我们首先根据二元神经网络的独特性精心设计了一个搜索空间。然后，我们提出了三种训练策略，它们大大增强了超级网络的训练效果，并提高了所有子网的性能。我们发现的二进制模型族在 20M 到 200M 的各种操作 (OP) 中的表现都优于之前的 BNN。例如，在 ImageNet 上，我们仅用 5700 万次操作就达到了 68.20% 的最高准确率。此外，我们还在物体检测任务中验证了这些搜索到的 BNN 的可移植性，我们的二进制检测器与搜索到的 BNN 一起在 MS COCO 数据集上实现了最先进的新结果，例如，在 370M OPs 的情况下，mAP 为 31.6%。源代码和模型将在 https://github.com/VDIGPKU/NAS-BNN 上发布。

{"title":"NAS-BNN: Neural Architecture Search for Binary Neural Networks","authors":"","doi":"10.1016/j.patcog.2024.111086","DOIUrl":"10.1016/j.patcog.2024.111086","url":null,"abstract":"<div><div>Binary Neural Networks (BNNs) have gained extensive attention for their superior inferencing efficiency and compression ratio compared to traditional full-precision networks. However, due to the unique characteristics of BNNs, designing a powerful binary architecture is challenging and often requires significant manpower. A promising solution is to utilize Neural Architecture Search (NAS) to assist in designing BNNs, but current NAS methods for BNNs are relatively straightforward and leave a performance gap between the searched models and manually designed ones. To address this gap, we propose a novel neural architecture search scheme for binary neural networks, named NAS-BNN. We first carefully design a search space based on the unique characteristics of BNNs. Then, we present three training strategies, which significantly enhance the training of supernet and boost the performance of all subnets. Our discovered binary model family outperforms previous BNNs for a wide range of operations (OPs) from 20M to 200M. For instance, we achieve 68.20% top-1 accuracy on ImageNet with only 57M OPs. In addition, we validate the transferability of these searched BNNs on the object detection task, and our binary detectors with the searched BNNs achieve a novel state-of-the-art result, <em>e.g.</em>, 31.6% mAP with 370M OPs, on MS COCO dataset. The source code and models will be released at <span><span>https://github.com/VDIGPKU/NAS-BNN</span><svg><path></path></svg></span>.</div></div>","PeriodicalId":49713,"journal":{"name":"Pattern Recognition","volume":null,"pages":null},"PeriodicalIF":7.5,"publicationDate":"2024-10-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142553016","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

MSCMNet: Multi-scale Semantic Correlation Mining for Visible-Infrared Person Re-Identification MSCMNet：用于可见光-红外线人员再识别的多尺度语义关联挖掘

IF 7.5 1区计算机科学 Q1 COMPUTER SCIENCE, ARTIFICIAL INTELLIGENCE

Pattern Recognition

Pub Date : 2024-10-23 DOI: 10.1016/j.patcog.2024.111090

The main challenge in the Visible-Infrared Person Re-Identification (VI-ReID) task lies in extracting discriminative features from different modalities for matching purposes. While existing studies primarily focus on reducing modal discrepancies, the modality information fails to be thoroughly exploited. To solve this problem, the Multi-scale Semantic Correlation Mining network (MSCMNet) is proposed to comprehensively exploit semantic features at multiple scales. The network fuses shallow-level features into the deep network through dimensionality reduction and mapping, and the fused features are utilized to minimize modality information loss in feature extraction. Firstly, considering the effective utilization of modality information, the Multi-scale Information Correlation Mining Block (MIMB) is designed to fuse features at different scales and explore the semantic correlation of fusion features. Secondly, in order to enrich the semantic information that MIMB can utilize, the Quadruple-stream Feature Extractor (QFE) with non-shared parameters is specifically designed to extract information from different dimensions of the dataset. Finally, the Quadruple Center Triplet Loss (QCT) is further proposed to address the information discrepancy in the comprehensive features. Extensive experiments on the SYSU-MM01, RegDB, and LLCM datasets demonstrate that the proposed MSCMNet achieves the greatest accuracy. We release the source code on https://github.com/Hua-XC/MSCMNet.

可见光-红外人员再识别（VI-ReID）任务的主要挑战在于从不同模态中提取辨别特征用于匹配。虽然现有研究主要侧重于减少模态差异，但模态信息未能得到彻底利用。为了解决这个问题，我们提出了多尺度语义关联挖掘网络（MSCMNet），以全面利用多个尺度的语义特征。该网络通过降维和映射将浅层特征融合到深层网络中，并利用融合后的特征在特征提取中尽量减少模态信息损失。首先，考虑到模态信息的有效利用，设计了多尺度信息相关性挖掘模块（MIMB）来融合不同尺度的特征，探索融合特征的语义相关性。其次，为了丰富 MIMB 可利用的语义信息，专门设计了具有非共享参数的四重流特征提取器（QFE），以提取数据集不同维度的信息。最后，还进一步提出了四重中心三重丢失（QCT）来解决综合特征中的信息差异问题。在 SYSU-MM01、RegDB 和 LLCM 数据集上进行的大量实验表明，所提出的 MSCMNet 实现了最高的准确率。我们在 https://github.com/Hua-XC/MSCMNet 上发布了源代码。

{"title":"MSCMNet: Multi-scale Semantic Correlation Mining for Visible-Infrared Person Re-Identification","authors":"","doi":"10.1016/j.patcog.2024.111090","DOIUrl":"10.1016/j.patcog.2024.111090","url":null,"abstract":"<div><div>The main challenge in the Visible-Infrared Person Re-Identification (VI-ReID) task lies in extracting discriminative features from different modalities for matching purposes. While existing studies primarily focus on reducing modal discrepancies, the modality information fails to be thoroughly exploited. To solve this problem, the Multi-scale Semantic Correlation Mining network (MSCMNet) is proposed to comprehensively exploit semantic features at multiple scales. The network fuses shallow-level features into the deep network through dimensionality reduction and mapping, and the fused features are utilized to minimize modality information loss in feature extraction. Firstly, considering the effective utilization of modality information, the Multi-scale Information Correlation Mining Block (MIMB) is designed to fuse features at different scales and explore the semantic correlation of fusion features. Secondly, in order to enrich the semantic information that MIMB can utilize, the Quadruple-stream Feature Extractor (QFE) with non-shared parameters is specifically designed to extract information from different dimensions of the dataset. Finally, the Quadruple Center Triplet Loss (QCT) is further proposed to address the information discrepancy in the comprehensive features. Extensive experiments on the SYSU-MM01, RegDB, and LLCM datasets demonstrate that the proposed MSCMNet achieves the greatest accuracy. We release the source code on <span><span>https://github.com/Hua-XC/MSCMNet</span><svg><path></path></svg></span>.</div></div>","PeriodicalId":49713,"journal":{"name":"Pattern Recognition","volume":null,"pages":null},"PeriodicalIF":7.5,"publicationDate":"2024-10-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142553025","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

One point is all you need for weakly supervised object detection 弱监督物体检测只需一个点

IF 7.5 1区计算机科学 Q1 COMPUTER SCIENCE, ARTIFICIAL INTELLIGENCE

Pattern Recognition

Pub Date : 2024-10-23 DOI: 10.1016/j.patcog.2024.111087

Object detection with weak annotations has attracted much attention recently. Weakly supervised object detection(WSOD) methods which only use image-level labels to train a detector encounter some severe problems that it cannot cover the whole object and the region proposal methods waste a large amount of time. Meanwhile, point supervised object detection(PSOD) leverages point annotations that remarkably improves the performance. However, point annotation is still complex and increase the costs of annotation. To overcome these issues, we propose a novel method which only requires one point per category for a training image. Compared to point annotation, our method significantly reduces the annotation cost as the number of point annotations is largely reduced. We design a framework to train a detector with one point per category annotation. Firstly, a pseudo box generation module is introduced to generate the corresponding pseudo boxes of the annotated points. Then, inspired by the observation that the features of objects with the same class in an image are very similar, a dense instances mining module is proposed to make use of the similarity between the features of objects with the same class to discover unlabeled instances and generate pseudo category heatmaps. Finally, the pseudo boxes and pseudo category heatmaps are leveraged to train a detector. Experiments conducted on popular open-source datasets verify the effectiveness of our annotation method and framework. Our proposed method outperforms previous WSOD methods and achieves comparable performance with some PSOD methods in a more efficient way.

近来，弱注释物体检测备受关注。弱监督物体检测（WSOD）方法仅使用图像级标签来训练检测器，会遇到一些严重的问题，即无法覆盖整个物体，而且区域建议方法会浪费大量时间。与此同时，点监督对象检测（PSOD）利用点注释显著提高了性能。然而，点标注仍然很复杂，而且会增加标注成本。为了克服这些问题，我们提出了一种新方法，它只需要对训练图像的每个类别标注一个点。与点标注相比，我们的方法大大减少了点标注的数量，从而显著降低了标注成本。我们设计了一个框架，用每个类别一个点的注释来训练检测器。首先，我们引入了一个伪方框生成模块，用于生成注释点对应的伪方框。然后，受图像中同一类别的物体特征非常相似这一观察结果的启发，我们提出了一个密集实例挖掘模块，利用同一类别的物体特征之间的相似性来发现未标注的实例，并生成伪类别热图。最后，利用伪方框和伪类别热图来训练检测器。在流行的开源数据集上进行的实验验证了我们的标注方法和框架的有效性。我们提出的方法优于之前的 WSOD 方法，并以更高效的方式实现了与某些 PSOD 方法相当的性能。

{"title":"One point is all you need for weakly supervised object detection","authors":"","doi":"10.1016/j.patcog.2024.111087","DOIUrl":"10.1016/j.patcog.2024.111087","url":null,"abstract":"<div><div>Object detection with weak annotations has attracted much attention recently. Weakly supervised object detection(WSOD) methods which only use image-level labels to train a detector encounter some severe problems that it cannot cover the whole object and the region proposal methods waste a large amount of time. Meanwhile, point supervised object detection(PSOD) leverages point annotations that remarkably improves the performance. However, point annotation is still complex and increase the costs of annotation. To overcome these issues, we propose a novel method which only requires one point per category for a training image. Compared to point annotation, our method significantly reduces the annotation cost as the number of point annotations is largely reduced. We design a framework to train a detector with one point per category annotation. Firstly, a pseudo box generation module is introduced to generate the corresponding pseudo boxes of the annotated points. Then, inspired by the observation that the features of objects with the same class in an image are very similar, a dense instances mining module is proposed to make use of the similarity between the features of objects with the same class to discover unlabeled instances and generate pseudo category heatmaps. Finally, the pseudo boxes and pseudo category heatmaps are leveraged to train a detector. Experiments conducted on popular open-source datasets verify the effectiveness of our annotation method and framework. Our proposed method outperforms previous WSOD methods and achieves comparable performance with some PSOD methods in a more efficient way.</div></div>","PeriodicalId":49713,"journal":{"name":"Pattern Recognition","volume":null,"pages":null},"PeriodicalIF":7.5,"publicationDate":"2024-10-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142553049","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0