Neurocomputing最新文献_第6页

A survey of graph neural networks and their industrial applications 图神经网络及其工业应用概览

IF 5.5 2区计算机科学 Q1 COMPUTER SCIENCE, ARTIFICIAL INTELLIGENCE

Neurocomputing

Pub Date : 2024-10-28 DOI: 10.1016/j.neucom.2024.128761

Haoran Lu , Lei Wang , Xiaoliang Ma , Jun Cheng , Mengchu Zhou

Graph Neural Networks (GNNs) have emerged as a powerful tool for analyzing and modeling graph-structured data. In recent years, GNNs have gained significant attention in various domains. This review paper aims to provide an overview of the state-of-the-art graph neural network techniques and their industrial applications. First, we introduce the fundamental concepts and architectures of GNNs, highlighting their ability to capture complex relationships and dependencies in graph data. We then delve into the variants and evolution of graphs, including directed graphs, heterogeneous graphs, dynamic graphs, and hypergraphs. Next, we discuss the interpretability of GNN, and GNN theory including graph augmentation, expressivity, and over-smoothing. Finally, we introduce the specific use cases of GNNs in industrial settings, including finance, biology, knowledge graphs, recommendation systems, Internet of Things (IoT), and knowledge distillation. This review paper highlights the immense potential of GNNs in solving real-world problems, while also addressing the challenges and opportunities for further advancement in this field.

图神经网络（GNN）已成为分析和模拟图结构数据的强大工具。近年来，图神经网络在各个领域都获得了极大的关注。本综述旨在概述最先进的图神经网络技术及其工业应用。首先，我们将介绍图神经网络的基本概念和架构，强调其捕捉图数据中复杂关系和依赖性的能力。然后，我们深入探讨图的变体和演变，包括有向图、异构图、动态图和超图。接下来，我们将讨论 GNN 的可解释性以及 GNN 理论，包括图增强、表现力和过度平滑。最后，我们介绍了 GNN 在工业环境中的具体用例，包括金融、生物、知识图谱、推荐系统、物联网 (IoT) 和知识提炼。本综述论文强调了 GNN 在解决现实世界问题方面的巨大潜力，同时也探讨了该领域进一步发展所面临的挑战和机遇。

{"title":"A survey of graph neural networks and their industrial applications","authors":"Haoran Lu , Lei Wang , Xiaoliang Ma , Jun Cheng , Mengchu Zhou","doi":"10.1016/j.neucom.2024.128761","DOIUrl":"10.1016/j.neucom.2024.128761","url":null,"abstract":"<div><div>Graph Neural Networks (GNNs) have emerged as a powerful tool for analyzing and modeling graph-structured data. In recent years, GNNs have gained significant attention in various domains. This review paper aims to provide an overview of the state-of-the-art graph neural network techniques and their industrial applications. First, we introduce the fundamental concepts and architectures of GNNs, highlighting their ability to capture complex relationships and dependencies in graph data. We then delve into the variants and evolution of graphs, including directed graphs, heterogeneous graphs, dynamic graphs, and hypergraphs. Next, we discuss the interpretability of GNN, and GNN theory including graph augmentation, expressivity, and over-smoothing. Finally, we introduce the specific use cases of GNNs in industrial settings, including finance, biology, knowledge graphs, recommendation systems, Internet of Things (IoT), and knowledge distillation. This review paper highlights the immense potential of GNNs in solving real-world problems, while also addressing the challenges and opportunities for further advancement in this field.</div></div>","PeriodicalId":19268,"journal":{"name":"Neurocomputing","volume":"614 ","pages":"Article 128761"},"PeriodicalIF":5.5,"publicationDate":"2024-10-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142586922","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Multi-attention associate prediction network for visual tracking 用于视觉跟踪的多注意力关联预测网络

IF 5.5 2区计算机科学 Q1 COMPUTER SCIENCE, ARTIFICIAL INTELLIGENCE

Neurocomputing

Pub Date : 2024-10-28 DOI: 10.1016/j.neucom.2024.128785

Xinglong Sun , Haijiang Sun , Shan Jiang , Jiacheng Wang , Xilai Wei , Zhonghe Hu

Classification-regression prediction networks have realized impressive success in several modern deep trackers. However, there is an inherent difference between classification and regression tasks, so they have diverse even opposite demands for feature matching. Existed models always ignore the key issue and only employ a unified matching block in two task branches, decaying the decision quality. Besides, these models also struggle with decision misalignment situation. In this paper, we propose a multi-attention associate prediction network (MAPNet) to tackle the above problems. Concretely, two novel matchers, i.e., category-aware matcher and spatial-aware matcher, are first designed for feature comparison by integrating self, cross, channel or spatial attentions organically. They are capable of fully capturing the category-related semantics for classification and the local spatial contexts for regression, respectively. Then, we present a dual alignment module to enhance the correspondences between two branches, which is useful to find the optimal tracking solution. Finally, we describe a Siamese tracker built upon the proposed prediction network, which achieves the leading performance on five tracking benchmarks, consisting of LaSOT, TrackingNet, GOT-10k, TNL2k and UAV123, and surpasses other state-of-the-art approaches.

分类-回归预测网络在一些现代深度跟踪器中取得了令人瞩目的成功。然而，分类任务和回归任务之间存在着内在差异，因此它们对特征匹配的要求各不相同，甚至截然相反。现有模型总是忽略这一关键问题，只在两个任务分支中采用统一的匹配块，从而降低了决策质量。此外，这些模型还难以解决决策错位的问题。本文提出了一种多注意力关联预测网络（MAPNet）来解决上述问题。具体来说，我们首先设计了两个新型匹配器，即类别感知匹配器和空间感知匹配器，通过有机整合自身、交叉、渠道或空间注意力来进行特征比较。它们分别能够充分捕捉用于分类的类别相关语义和用于回归的局部空间上下文。然后，我们提出了一个双对齐模块，以增强两个分支之间的对应关系，这有助于找到最佳跟踪方案。最后，我们介绍了基于所提预测网络的连体跟踪器，该跟踪器在 LaSOT、TrackingNet、GOT-10k、TNL2k 和 UAV123 等五个跟踪基准测试中取得了领先的性能，并超越了其他最先进的方法。

{"title":"Multi-attention associate prediction network for visual tracking","authors":"Xinglong Sun , Haijiang Sun , Shan Jiang , Jiacheng Wang , Xilai Wei , Zhonghe Hu","doi":"10.1016/j.neucom.2024.128785","DOIUrl":"10.1016/j.neucom.2024.128785","url":null,"abstract":"<div><div>Classification-regression prediction networks have realized impressive success in several modern deep trackers. However, there is an inherent difference between classification and regression tasks, so they have diverse even opposite demands for feature matching. Existed models always ignore the key issue and only employ a unified matching block in two task branches, decaying the decision quality. Besides, these models also struggle with decision misalignment situation. In this paper, we propose a multi-attention associate prediction network (MAPNet) to tackle the above problems. Concretely, two novel matchers, i.e., category-aware matcher and spatial-aware matcher, are first designed for feature comparison by integrating self, cross, channel or spatial attentions organically. They are capable of fully capturing the category-related semantics for classification and the local spatial contexts for regression, respectively. Then, we present a dual alignment module to enhance the correspondences between two branches, which is useful to find the optimal tracking solution. Finally, we describe a Siamese tracker built upon the proposed prediction network, which achieves the leading performance on five tracking benchmarks, consisting of LaSOT, TrackingNet, GOT-10k, TNL2k and UAV123, and surpasses other state-of-the-art approaches.</div></div>","PeriodicalId":19268,"journal":{"name":"Neurocomputing","volume":"614 ","pages":"Article 128785"},"PeriodicalIF":5.5,"publicationDate":"2024-10-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142573298","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Simulation-based effective comparative analysis of neuron circuits for neuromorphic computation systems 基于仿真的神经元电路有效比较分析，用于神经形态计算系统

IF 5.5 2区计算机科学 Q1 COMPUTER SCIENCE, ARTIFICIAL INTELLIGENCE

Neurocomputing

Pub Date : 2024-10-28 DOI: 10.1016/j.neucom.2024.128758

Deepthi M.S. , Shashidhara H.R. , Jayaramu Raghu , Rudraswamy S.B.

The spiking neural networks (SNN) that are inspired by the human brain offers wider scope for application in the growth of neuromorphic computing systems due to their brain level computational capabilities, reduced power consumption, and minimal data movement cost, among other advantages. Spike-based neurons and synapses are the essential building blocks of SNN, and their efficient implementation is vital to their performance enhancement. In this regard, the design and implementation of spiking neurons have been the major focus among the researchers. In this paper, functioning of different leaky integrate fire (LIF)-based spiking neuron circuits like frequency adaptable CMOS-based LIF, resistor-capacitor-based (RC) LIF, and volatile memristor-based LIF are subjected to comparison. The work mainly focuses on revealing analysis of spike duration and amplitude, number of spikes produced during excitation period, threshold operation, field of application, and various other significant parameters of aforementioned neuron circuits. Extensive simulations of these circuits are carried out utilizing the Cadence Virtuoso simulation environment in order to validate their behavior. Further, a brief comparative analysis is executed considering into account the attributes like circuit complexity, supply voltage, firing rate, membrane capacitance, nature of input/output, refractory mechanism, and energy consumption per spike. This work seeks to assist researchers in selecting an appropriate LIF model to efficiently construct memristors and/or non-memristors based SNN for certain application.

尖峰神经网络（SNN）的灵感来源于人脑，具有脑级计算能力、功耗低、数据移动成本小等优点，为神经形态计算系统的发展提供了更广阔的应用空间。基于尖峰的神经元和突触是 SNN 的基本构件，它们的高效实现对提高 SNN 的性能至关重要。在这方面，尖峰神经元的设计和实现一直是研究人员关注的重点。本文比较了基于不同漏电集成火花（LIF）的尖峰神经元电路的功能，如基于 CMOS 的频率自适应 LIF、基于电阻电容（RC）的 LIF 和基于易失性 Memristor 的 LIF。工作主要集中在对上述神经元电路的尖峰持续时间和振幅、激励期间产生的尖峰数量、阈值操作、应用领域和其他各种重要参数进行揭示分析。研究利用 Cadence Virtuoso 仿真环境对这些电路进行了大量仿真，以验证其行为。此外，考虑到电路复杂性、电源电压、发射率、膜电容、输入/输出性质、耐火机制和每个尖峰的能耗等属性，还进行了简要的比较分析。这项工作旨在帮助研究人员选择合适的 LIF 模型，以便针对特定应用高效地构建基于忆阻器和/或非忆阻器的 SNN。

{"title":"Simulation-based effective comparative analysis of neuron circuits for neuromorphic computation systems","authors":"Deepthi M.S. , Shashidhara H.R. , Jayaramu Raghu , Rudraswamy S.B.","doi":"10.1016/j.neucom.2024.128758","DOIUrl":"10.1016/j.neucom.2024.128758","url":null,"abstract":"<div><div>The spiking neural networks (SNN) that are inspired by the human brain offers wider scope for application in the growth of neuromorphic computing systems due to their brain level computational capabilities, reduced power consumption, and minimal data movement cost, among other advantages. Spike-based neurons and synapses are the essential building blocks of SNN, and their efficient implementation is vital to their performance enhancement. In this regard, the design and implementation of spiking neurons have been the major focus among the researchers. In this paper, functioning of different leaky integrate fire (LIF)-based spiking neuron circuits like frequency adaptable CMOS-based LIF, resistor-capacitor-based (RC) LIF, and volatile memristor-based LIF are subjected to comparison. The work mainly focuses on revealing analysis of spike duration and amplitude, number of spikes produced during excitation period, threshold operation, field of application, and various other significant parameters of aforementioned neuron circuits. Extensive simulations of these circuits are carried out utilizing the Cadence Virtuoso simulation environment in order to validate their behavior. Further, a brief comparative analysis is executed considering into account the attributes like circuit complexity, supply voltage, firing rate, membrane capacitance, nature of input/output, refractory mechanism, and energy consumption per spike. This work seeks to assist researchers in selecting an appropriate LIF model to efficiently construct memristors and/or non-memristors based SNN for certain application.</div></div>","PeriodicalId":19268,"journal":{"name":"Neurocomputing","volume":"614 ","pages":"Article 128758"},"PeriodicalIF":5.5,"publicationDate":"2024-10-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142573219","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

A review of AI edge devices and lightweight CNN and LLM deployment 人工智能边缘设备及轻量级 CNN 和 LLM 部署回顾

IF 5.5 2区计算机科学 Q1 COMPUTER SCIENCE, ARTIFICIAL INTELLIGENCE

Neurocomputing

Pub Date : 2024-10-28 DOI: 10.1016/j.neucom.2024.128791

Kailai Sun , Xinwei Wang , Xi Miao , Qianchuan Zhao

Artificial Intelligence of Things (AIoT) which integrates artificial intelligence (AI) and the Internet of Things (IoT), has attracted increasing attention recently. With the remarkable development of AI, convolutional neural networks (CNN) have achieved great success from research to deployment in many applications. However, deploying complex and state-of-the-art (SOTA) AI models on edge applications is increasingly a big challenge. This paper investigates literature that deploys lightweight CNNs on AI edge devices in practice. We provide a comprehensive analysis of them and many practical suggestions for researchers: how to obtain/design lightweight CNNs, select suitable AI edge devices, and compress and deploy them in practice. Finally, future trends and opportunities are presented, including the deployment of large language models, trustworthy AI and robust deployment.

将人工智能（AI）和物联网（IoT）融为一体的人工智能物联网（AIoT）近来日益受到关注。随着人工智能的显著发展，卷积神经网络（CNN）从研究到部署在许多应用中都取得了巨大成功。然而，在边缘应用中部署复杂而先进（SOTA）的人工智能模型正日益成为一个巨大的挑战。本文研究了在人工智能边缘设备上实际部署轻量级 CNN 的文献。我们对它们进行了全面分析，并为研究人员提供了许多实用建议：如何获取/设计轻量级 CNN、选择合适的人工智能边缘设备，以及在实践中压缩和部署它们。最后，我们介绍了未来的趋势和机遇，包括大型语言模型的部署、可信的人工智能和稳健的部署。

引用次数: 0

IoU-guided Siamese network with high-confidence template fusion for visual tracking 用于视觉跟踪的高置信度模板融合 IoU 引导连体网络

IF 5.5 2区计算机科学 Q1 COMPUTER SCIENCE, ARTIFICIAL INTELLIGENCE

Neurocomputing

Pub Date : 2024-10-28 DOI: 10.1016/j.neucom.2024.128774

Zhigang Liu , Hao Huang , Hongyu Dong , Fuyuan Xing

Existing IoU-guided trackers use IoU score to weight the classification score only in testing phase, this model mismatch between training and testing phases leads to poor tracking performance especially when facing background distractors. In this paper, we propose an IoU-guided Siamese network with High-confidence template fusion (SiamIH) for visual tracking. An IoU-guided distractor suppression network is proposed, which uses IoU information to guide classification in training phase and testing phase, and makes the tracking model to suppress background distractors. To cope with appearance variations, we design a high-confidence template fusion network that fuses APCE-based high-confidence template and the initial template to generate more reliable template. Experimental results on OTB2013, OTB2015, UAV123, LaSOT, and GOT10k demonstrate that the proposed SiamIH achieves state-of-the-art tracking performance.

现有的 IoU 引导跟踪器仅在测试阶段使用 IoU 分数作为分类分数的权重，这种训练和测试阶段的模型不匹配会导致跟踪性能低下，尤其是在面对背景干扰时。在本文中，我们提出了一种用于视觉跟踪的高置信度模板融合 IoU 引导暹罗网络（SiamIH）。该网络在训练和测试阶段利用 IoU 信息指导分类，并使跟踪模型抑制背景干扰。为了应对外观变化，我们设计了高置信度模板融合网络，将基于 APCE 的高置信度模板与初始模板融合，生成更可靠的模板。在 OTB2013、OTB2015、UAV123、LaSOT 和 GOT10k 上的实验结果表明，所提出的 SiamIH 实现了最先进的跟踪性能。

引用次数: 0

Auditing privacy budget of differentially private neural network models 审核不同隐私神经网络模型的隐私预算

IF 5.5 2区计算机科学 Q1 COMPUTER SCIENCE, ARTIFICIAL INTELLIGENCE

Neurocomputing

Pub Date : 2024-10-28 DOI: 10.1016/j.neucom.2024.128756

Wen Huang , Zhishuo Zhang , Weixin Zhao , Jian Peng , Wenzheng Xu , Yongjian Liao , Shijie Zhou , Ziming Wang

In recent years, neural network models are used in various tasks. To eliminate privacy concern, differential privacy (DP) is introduced to the training phase of neural network models. However, introducing DP into neural network models is very subtle and error-prone, resulting in that some differentially private neural network models may not achieve privacy guarantee claimed. In this paper, we propose a method, which can audit privacy budget of differentially private neural network models. The proposed method is general and can be used to audit some other AI models. We elaborate on how to audit privacy budget of basic DP mechanisms and neural network models by the proposed method first. Then, we run experiments to verify our method. Experiment results indicate that the proposed method is better than the advanced method and the auditing precise is high when the privacy budget is small. In particular, when auditing privacy budget of ResNet-18 over CIFAR-10 protected by the differentially private mechanism with theoretical privacy budget 0.2, the accuracy of our method is about 17 times that of the state-of-the-art method. For the simpler dataset FMNIST, the accuracy of our method is about 32 times that of the state-of-the-art method when theoretical privacy budget is 0.2.

近年来，神经网络模型被广泛应用于各种任务中。为了消除对隐私的担忧，人们在神经网络模型的训练阶段引入了差分隐私（DP）。然而，在神经网络模型中引入 DP 非常微妙且容易出错，导致一些差异化隐私神经网络模型可能无法实现所声称的隐私保证。在本文中，我们提出了一种方法，可以审核差异化隐私神经网络模型的隐私预算。本文提出的方法具有通用性，可用于审核其他人工智能模型。我们首先阐述了如何利用所提出的方法审核基本 DP 机制和神经网络模型的隐私预算。然后，我们通过实验来验证我们的方法。实验结果表明，当隐私预算较小时，建议的方法优于先进的方法，且审计精度较高。特别是，当对 ResNet-18 的隐私预算进行审计时，在理论隐私预算为 0.2 的情况下，在 CIFAR-10 上使用差异化隐私机制保护的 ResNet-18，我们的方法的精确度约为先进方法的 17 倍。对于更简单的数据集 FMNIST，当理论隐私预算为 0.2 时，我们的方法的准确率约为最新方法的 32 倍。

{"title":"Auditing privacy budget of differentially private neural network models","authors":"Wen Huang , Zhishuo Zhang , Weixin Zhao , Jian Peng , Wenzheng Xu , Yongjian Liao , Shijie Zhou , Ziming Wang","doi":"10.1016/j.neucom.2024.128756","DOIUrl":"10.1016/j.neucom.2024.128756","url":null,"abstract":"<div><div>In recent years, neural network models are used in various tasks. To eliminate privacy concern, differential privacy (DP) is introduced to the training phase of neural network models. However, introducing DP into neural network models is very subtle and error-prone, resulting in that some differentially private neural network models may not achieve privacy guarantee claimed. In this paper, we propose a method, which can audit privacy budget of differentially private neural network models. The proposed method is general and can be used to audit some other AI models. We elaborate on how to audit privacy budget of basic DP mechanisms and neural network models by the proposed method first. Then, we run experiments to verify our method. Experiment results indicate that the proposed method is better than the advanced method and the auditing precise is high when the privacy budget is small. In particular, when auditing privacy budget of ResNet-18 over CIFAR-10 protected by the differentially private mechanism with theoretical privacy budget 0.2, the accuracy of our method is about 17 times that of the state-of-the-art method. For the simpler dataset FMNIST, the accuracy of our method is about 32 times that of the state-of-the-art method when theoretical privacy budget is 0.2.</div></div>","PeriodicalId":19268,"journal":{"name":"Neurocomputing","volume":"614 ","pages":"Article 128756"},"PeriodicalIF":5.5,"publicationDate":"2024-10-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142578802","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

ERAT-DLoRA: Parameter-efficient tuning with enhanced range adaptation in time and depth aware dynamic LoRA ERAT-DLoRA：在时间和深度感知动态 LoRA 中通过增强范围自适应进行参数高效调整

IF 5.5 2区计算机科学 Q1 COMPUTER SCIENCE, ARTIFICIAL INTELLIGENCE

Neurocomputing

Pub Date : 2024-10-28 DOI: 10.1016/j.neucom.2024.128778

Dan Luo , Kangfeng Zheng , Chunhua Wu , Xiujuan Wang , Jvjie Wang

Despite their potential, the industrial deployment of large language models (LLMs) is constrained by traditional fine-tuning procedures that are both resource-intensive and time-consuming. Low-Rank Adaptation (LoRA) has emerged as a pioneering methodology for addressing these challenges. By integrating low-rank decomposition matrices into network weights to reduce trainable parameters, LoRA effectively accelerates the adaptation process. While research on LoRA primarily focuses on adjusting low-rank matrices, DyLoRA optimizes the rank-setting mechanism to avoid extensive effort in rank size training and searching. However, DyLoRA rank configuration mechanism has its own limitation. First, DyLoRA sets the same rank size for all the low-rank adaptation layers at each time step. Given that layers with different depth contain distinct information, they should have varying rank values to accurately capture their unique characteristics. Second, the truncated phase selected for ordering representation based on nested dropout regulation is only half dynamic, continuously dropping tail units, thereby limiting its ability to access information. In this work, we propose a novel technique, enhanced range adaptation in time and depth aware dynamic LoRA (ERAT-DLoRA) to address these problems. The ERAT-DLoRA method introduces a dynamic range to the truncated phase that makes the truncated phase fully dynamic. Additionally, we design a time and layer-aware dynamic rank to ensure appropriate adjustments at different time steps and layer levels. We evaluate our solution on natural languages understanding and language generation tasks. Extensive evaluation results demonstrate the effectiveness of the proposed method.

尽管大型语言模型（LLMs）潜力巨大，但其工业化部署却受到传统微调程序的限制，这些程序既耗费资源又耗费时间。低阶适应（Low-Rank Adaptation，LoRA）已成为应对这些挑战的开创性方法。通过将低阶分解矩阵整合到网络权重中以减少可训练参数，LoRA 有效地加快了适应过程。有关 LoRA 的研究主要集中在调整低秩矩阵上，而 DyLoRA 则优化了秩设置机制，避免了秩大小训练和搜索的大量工作。然而，DyLoRA 的秩配置机制有其自身的局限性。首先，DyLoRA 在每个时间步为所有低秩适应层设置相同的秩大小。鉴于不同深度的层包含不同的信息，它们应该有不同的秩值，以准确捕捉其独特的特征。其次，根据嵌套丢弃调节为排序表示所选择的截断相位只有一半是动态的，会不断丢弃尾部单元，从而限制了其获取信息的能力。在这项工作中，我们提出了一种新技术--时间和深度感知动态 LoRA（ERAT-DLoRA）中的增强范围适应，以解决这些问题。ERAT-DLoRA 方法为截断阶段引入了动态范围，使截断阶段完全动态化。此外，我们还设计了一种时间和层感知动态等级，以确保在不同的时间步骤和层级进行适当的调整。我们在自然语言理解和语言生成任务中评估了我们的解决方案。广泛的评估结果证明了所提方法的有效性。

{"title":"ERAT-DLoRA: Parameter-efficient tuning with enhanced range adaptation in time and depth aware dynamic LoRA","authors":"Dan Luo , Kangfeng Zheng , Chunhua Wu , Xiujuan Wang , Jvjie Wang","doi":"10.1016/j.neucom.2024.128778","DOIUrl":"10.1016/j.neucom.2024.128778","url":null,"abstract":"<div><div>Despite their potential, the industrial deployment of large language models (LLMs) is constrained by traditional fine-tuning procedures that are both resource-intensive and time-consuming. Low-Rank Adaptation (LoRA) has emerged as a pioneering methodology for addressing these challenges. By integrating low-rank decomposition matrices into network weights to reduce trainable parameters, LoRA effectively accelerates the adaptation process. While research on LoRA primarily focuses on adjusting low-rank matrices, DyLoRA optimizes the rank-setting mechanism to avoid extensive effort in rank size training and searching. However, DyLoRA rank configuration mechanism has its own limitation. First, DyLoRA sets the same rank size for all the low-rank adaptation layers at each time step. Given that layers with different depth contain distinct information, they should have varying rank values to accurately capture their unique characteristics. Second, the truncated phase selected for ordering representation based on nested dropout regulation is only half dynamic, continuously dropping tail units, thereby limiting its ability to access information. In this work, we propose a novel technique, enhanced range adaptation in time and depth aware dynamic LoRA (ERAT-DLoRA) to address these problems. The ERAT-DLoRA method introduces a dynamic range to the truncated phase that makes the truncated phase fully dynamic. Additionally, we design a time and layer-aware dynamic rank to ensure appropriate adjustments at different time steps and layer levels. We evaluate our solution on natural languages understanding and language generation tasks. Extensive evaluation results demonstrate the effectiveness of the proposed method.</div></div>","PeriodicalId":19268,"journal":{"name":"Neurocomputing","volume":"614 ","pages":"Article 128778"},"PeriodicalIF":5.5,"publicationDate":"2024-10-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142586926","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Subclass consistency regularization for learning with noisy labels based on contrastive learning 基于对比学习的噪声标签学习的子类一致性正则化

IF 5.5 2区计算机科学 Q1 COMPUTER SCIENCE, ARTIFICIAL INTELLIGENCE

Neurocomputing

Pub Date : 2024-10-28 DOI: 10.1016/j.neucom.2024.128759

Xinkai Sun, Sanguo Zhang

A prominent effect of label noise on neural networks is the disruption of the consistency of predictions. While prior efforts primarily focused on predictions’ consistency at the individual instance level, they often fell short of fully harnessing the consistency across multiple instances. This paper introduces subclass consistency regularization (SCCR) to maximize the potential of this collective consistency of predictions. SCCR mitigates the impact of label noise on neural networks by imposing constraints on the consistency of predictions within each subclass. However, constructing high-quality subclasses poses a formidable challenge, which we formulate as a special clustering problem. To efficiently establish these subclasses, we incorporate a clustering-based contrastive learning framework. Additionally, we introduce the

Q

-enhancing algorithm to tailor the contrastive learning framework, ensuring alignment with subclass construction. We conducted comprehensive experiments using benchmark datasets and real datasets to evaluate the effectiveness of our proposed method under various scenarios with differing noise rates. The results unequivocally demonstrate the enhancement in classification accuracy, especially in challenging high-noise settings. Moreover, the refined contrastive learning framework significantly elevates the quality of subclasses even in the presence of noise. Furthermore, we delve into the compatibility of contrastive learning and learning with noisy labels, using the projection head as an illustrative example. This investigation sheds light on an aspect that has hitherto been overlooked in prior research efforts.

标签噪声对神经网络的一个显著影响是破坏了预测的一致性。之前的研究主要关注单个实例层面的预测一致性，但往往无法充分利用多个实例之间的一致性。本文介绍了子类一致性正则化（SCCR），以最大限度地发挥这种集体一致性预测的潜力。SCCR 通过对每个子类内预测的一致性施加约束，减轻了标签噪声对神经网络的影响。然而，构建高质量的子类是一项艰巨的挑战，我们将其表述为一个特殊的聚类问题。为了有效地建立这些子类，我们采用了基于聚类的对比学习框架。此外，我们还引入了 Q 增强算法来调整对比学习框架，确保与子类构建保持一致。我们使用基准数据集和真实数据集进行了全面的实验，以评估我们提出的方法在不同噪声率的各种情况下的有效性。实验结果清楚地证明了分类准确率的提高，尤其是在具有挑战性的高噪声环境下。此外，即使在存在噪声的情况下，经过改进的对比学习框架也能显著提高子类的质量。此外，我们还以投影头为例，深入探讨了对比学习与噪声标签学习的兼容性。这项研究揭示了之前的研究中一直被忽视的一个方面。

{"title":"Subclass consistency regularization for learning with noisy labels based on contrastive learning","authors":"Xinkai Sun, Sanguo Zhang","doi":"10.1016/j.neucom.2024.128759","DOIUrl":"10.1016/j.neucom.2024.128759","url":null,"abstract":"<div><div>A prominent effect of label noise on neural networks is the disruption of the consistency of predictions. While prior efforts primarily focused on predictions’ consistency at the individual instance level, they often fell short of fully harnessing the consistency across multiple instances. This paper introduces subclass consistency regularization (SCCR) to maximize the potential of this collective consistency of predictions. SCCR mitigates the impact of label noise on neural networks by imposing constraints on the consistency of predictions within each subclass. However, constructing high-quality subclasses poses a formidable challenge, which we formulate as a special clustering problem. To efficiently establish these subclasses, we incorporate a clustering-based contrastive learning framework. Additionally, we introduce the <span><math><mi>Q</mi></math></span>-enhancing algorithm to tailor the contrastive learning framework, ensuring alignment with subclass construction. We conducted comprehensive experiments using benchmark datasets and real datasets to evaluate the effectiveness of our proposed method under various scenarios with differing noise rates. The results unequivocally demonstrate the enhancement in classification accuracy, especially in challenging high-noise settings. Moreover, the refined contrastive learning framework significantly elevates the quality of subclasses even in the presence of noise. Furthermore, we delve into the compatibility of contrastive learning and learning with noisy labels, using the projection head as an illustrative example. This investigation sheds light on an aspect that has hitherto been overlooked in prior research efforts.</div></div>","PeriodicalId":19268,"journal":{"name":"Neurocomputing","volume":"614 ","pages":"Article 128759"},"PeriodicalIF":5.5,"publicationDate":"2024-10-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142592861","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

MHEC: One-shot relational learning of knowledge graphs completion based on multi-hop information enhancement MHEC：基于多跳信息增强的知识图谱完成的一次性关系学习

IF 5.5 2区计算机科学 Q1 COMPUTER SCIENCE, ARTIFICIAL INTELLIGENCE

Neurocomputing

Pub Date : 2024-10-28 DOI: 10.1016/j.neucom.2024.128760

Ruixin Ma, Buyun Gao, Weihe Wang, Longfei Wang, Xiaoru Wang, Liang Zhao

With the wide application of knowledge graphs, knowledge graph completion has garnered increasing attention in recent years. However, we find that the long tail relation is more common in the KG. These relations typically do not have a large number of triples for training and are referred to as few-shot relations. The knowledge graph completion in the few-shot scenario is a major challenge currently. The current mainstream knowledge graph completion algorithms have the following drawbacks. The metric-based methods lack interpretability of results, while the algorithms based on path interaction are not suitable for few-shot scenarios and the availability of the model is limited in sparse knowledge graphs. In this paper, we propose a one-shot relational learning of knowledge graphs completion based on multi-hop information enhancement(MHEC). Firstly, MHEC extracts entity concepts from multi-hop paths to obtain task related entity concepts and filters out noisy neighbor attributes. Then, MHEC combines multi-hop path information between head and tail to represent entity pairs. Compared to previous completion methods that only consider structural features of entities, MHEC considers the reasoning logic between entity pairs, which not only includes structural features but also possesses rich semantic features. Next, MHEC introduces a reasoning process in the completion task to address the issues of lack of interpretability in the one-shot scenario. In addition, to improve completion and reasoning quality in sparse knowledge graphs, MHEC utilizes contrastive learning to enhance pre-training vector representations of entities and relations and proposes a matching processor that leverages the semantic information of pre-training vectors to assist the reasoning model in expanding the multi-hop paths. Experiments demonstrate that MHEC outperforms the state-of-the-art completion techniques on real-world datasets NELL-One and FB15k237-One.

近年来，随着知识图谱的广泛应用，知识图谱补全越来越受到关注。然而，我们发现长尾关系在知识图谱中更为常见。这些关系通常没有大量的三元组可供训练，被称为 "少量关系"。在少量关系的情况下完成知识图谱是当前的一大挑战。目前主流的知识图谱补全算法有以下缺点。基于度量的方法缺乏结果的可解释性，而基于路径交互的算法不适合少点场景，在稀疏知识图谱中模型的可用性有限。在本文中，我们提出了一种基于多跳信息增强（MHEC）的知识图完成的一次性关系学习方法。首先，MHEC 从多跳路径中提取实体概念，得到与任务相关的实体概念，并过滤掉有噪声的邻居属性。然后，MHEC 结合头尾之间的多跳路径信息来表示实体对。与以往只考虑实体结构特征的完成方法相比，MHEC 考虑了实体对之间的推理逻辑，这不仅包括结构特征，还具有丰富的语义特征。接下来，MHEC 在完成任务中引入了推理过程，以解决单次场景中缺乏可解释性的问题。此外，为了提高稀疏知识图谱的完成和推理质量，MHEC利用对比学习来增强实体和关系的预训练向量表示，并提出了一种匹配处理器，利用预训练向量的语义信息来辅助推理模型扩展多跳路径。实验证明，在实际数据集 NELL-One 和 FB15k237-One 上，MHEC 的表现优于最先进的补全技术。

{"title":"MHEC: One-shot relational learning of knowledge graphs completion based on multi-hop information enhancement","authors":"Ruixin Ma, Buyun Gao, Weihe Wang, Longfei Wang, Xiaoru Wang, Liang Zhao","doi":"10.1016/j.neucom.2024.128760","DOIUrl":"10.1016/j.neucom.2024.128760","url":null,"abstract":"<div><div>With the wide application of knowledge graphs, knowledge graph completion has garnered increasing attention in recent years. However, we find that the long tail relation is more common in the KG. These relations typically do not have a large number of triples for training and are referred to as few-shot relations. The knowledge graph completion in the few-shot scenario is a major challenge currently. The current mainstream knowledge graph completion algorithms have the following drawbacks. The metric-based methods lack interpretability of results, while the algorithms based on path interaction are not suitable for few-shot scenarios and the availability of the model is limited in sparse knowledge graphs. In this paper, we propose a one-shot relational learning of knowledge graphs completion based on multi-hop information enhancement(MHEC). Firstly, MHEC extracts entity concepts from multi-hop paths to obtain task related entity concepts and filters out noisy neighbor attributes. Then, MHEC combines multi-hop path information between head and tail to represent entity pairs. Compared to previous completion methods that only consider structural features of entities, MHEC considers the reasoning logic between entity pairs, which not only includes structural features but also possesses rich semantic features. Next, MHEC introduces a reasoning process in the completion task to address the issues of lack of interpretability in the one-shot scenario. In addition, to improve completion and reasoning quality in sparse knowledge graphs, MHEC utilizes contrastive learning to enhance pre-training vector representations of entities and relations and proposes a matching processor that leverages the semantic information of pre-training vectors to assist the reasoning model in expanding the multi-hop paths. Experiments demonstrate that MHEC outperforms the state-of-the-art completion techniques on real-world datasets NELL-One and FB15k237-One.</div></div>","PeriodicalId":19268,"journal":{"name":"Neurocomputing","volume":"614 ","pages":"Article 128760"},"PeriodicalIF":5.5,"publicationDate":"2024-10-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142573297","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Observer-based adaptive neural network event-triggered quantized control for active suspensions with actuator saturation 基于观测器的自适应神经网络事件触发量化控制，用于具有致动器饱和度的主动悬挂系统

IF 5.5 2区计算机科学 Q1 COMPUTER SCIENCE, ARTIFICIAL INTELLIGENCE

Neurocomputing

Pub Date : 2024-10-28 DOI: 10.1016/j.neucom.2024.128770

Tiechao Wang, Hongyang Zhang, Shuai Sui

This paper proposes an adaptive neural network event-triggered and quantized output feedback control scheme for quarter vehicle active suspensions with actuator saturation. The scheme uses neural networks to approximate the unknown parts of the active suspension. When the system states of the suspension are not entirely available, a state observer is designed to estimate the unknown states. The measurable system states, partially estimated observer states, neural network weights, and a filtered virtual control are sequentially event-triggered, quantified, and transmitted to the controller via in-vehicle networks. The problem of non-differentiable virtual control is solved using dynamic surface control technology in the backstepping quantized control design. Integrating a Gaussian error function and a first-order auxiliary subsystem compensates for the nonlinearity caused by asymmetric saturation. Theoretical analysis proves that all error signals of the closed-loop active suspension system are semi-globally uniformly ultimately bounded, and the Zeno phenomenon can be ruled out. Simulation results validate the effectiveness of the proposed control method.

本文针对致动器饱和的四分之一车辆主动悬架提出了一种自适应神经网络事件触发和量化输出反馈控制方案。该方案利用神经网络对主动悬架的未知部分进行近似。当悬架的系统状态不完全可用时，设计一个状态观测器来估计未知状态。可测量的系统状态、部分估计的观测器状态、神经网络权重和滤波虚拟控制依次被事件触发、量化，并通过车载网络传输到控制器。在反步进量化控制设计中，利用动态表面控制技术解决了无差别虚拟控制问题。整合高斯误差函数和一阶辅助子系统可补偿非对称饱和引起的非线性问题。理论分析证明，闭环主动悬架系统的所有误差信号都是半全局均匀终界的，可以排除芝诺现象。仿真结果验证了所提控制方法的有效性。

{"title":"Observer-based adaptive neural network event-triggered quantized control for active suspensions with actuator saturation","authors":"Tiechao Wang, Hongyang Zhang, Shuai Sui","doi":"10.1016/j.neucom.2024.128770","DOIUrl":"10.1016/j.neucom.2024.128770","url":null,"abstract":"<div><div>This paper proposes an adaptive neural network event-triggered and quantized output feedback control scheme for quarter vehicle active suspensions with actuator saturation. The scheme uses neural networks to approximate the unknown parts of the active suspension. When the system states of the suspension are not entirely available, a state observer is designed to estimate the unknown states. The measurable system states, partially estimated observer states, neural network weights, and a filtered virtual control are sequentially event-triggered, quantified, and transmitted to the controller via in-vehicle networks. The problem of non-differentiable virtual control is solved using dynamic surface control technology in the backstepping quantized control design. Integrating a Gaussian error function and a first-order auxiliary subsystem compensates for the nonlinearity caused by asymmetric saturation. Theoretical analysis proves that all error signals of the closed-loop active suspension system are semi-globally uniformly ultimately bounded, and the Zeno phenomenon can be ruled out. Simulation results validate the effectiveness of the proposed control method.</div></div>","PeriodicalId":19268,"journal":{"name":"Neurocomputing","volume":"614 ","pages":"Article 128770"},"PeriodicalIF":5.5,"publicationDate":"2024-10-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142573193","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0