Yujie Wu, Siyuan Xu, Jibin Wu, Lei Deng, Mingkun Xu, Qinghao Wen, Guoqi Li
The Forward-Forward (FF) algorithm was recently proposed as a local learning method to address the limitations of backpropagation (BP), offering biological plausibility along with memory-efficient and highly parallelized computational benefits. However, it suffers from suboptimal performance and poor generalization, largely due to inadequate theoretical support and a lack of effective learning strategies. In this work, we reformulate FF using distance metric learning and propose a distance-forward algorithm (DF) to improve FF performance in supervised vision tasks while preserving its local computational properties, making it competitive for efficient on-chip learning. To achieve this, we reinterpret FF through the lens of centroid-based metric learning and develop a goodness-based N-pair margin loss to facilitate the learning of discriminative features. Furthermore, we integrate layer-collaboration local update strategies to reduce information loss caused by greedy local parameter updates. Our method surpasses existing FF models and other advanced local learning approaches, with accuracies of 99.7% on MNIST, 88.2% on CIFAR-10, 59% on CIFAR-100, 95.9% on SVHN, and 82.5% on ImageNette, respectively. Moreover, it achieves comparable performance with less than 40% memory cost compared to BP training, while exhibiting stronger robustness to multiple types of hardware-related noise, demonstrating its potential for online learning and energy-efficient computation on neuromorphic chips.
前向前馈（FF）算法是最近提出的一种局部学习方法，旨在解决反向传播（BP）的局限性，该算法不仅具有生物学上的合理性，还具有内存效率高、计算高度并行化等优点。然而，它的性能不理想，泛化能力差，这主要是由于理论支持不足和缺乏有效的学习策略。在这项工作中，我们使用距离度量学习重新表述了 FF，并提出了一种距离前向算法 (DF)，以提高 FF 在有监督视觉任务中的性能，同时保留其本地计算特性，使其在高效片上学习方面具有竞争力。为了实现这一目标，我们从基于中心点的度量学习角度重新解释了 FF，并开发了一种基于善度的 N 对边距损失，以促进区分性特征的学习。此外，我们还整合了层协作局部更新策略，以减少贪婪的局部参数更新造成的信息损失。我们的方法超越了现有的FF模型和其他先进的局部学习方法，在MNIST上的准确率为99.7%，在CIFAR-10上的准确率为88.2%，在CIFAR-100上的准确率为59%，在SVHN上的准确率为95.9%，在ImageNette上的准确率为82.5%。此外，与BP训练相比，它以不到40%的内存成本实现了可比的性能，同时对多种类型的硬件相关噪声表现出更强的鲁棒性，证明了它在神经形态芯片上的在线学习和节能计算潜力。
{"title":"Distance-Forward Learning: Enhancing the Forward-Forward Algorithm Towards High-Performance On-Chip Learning","authors":"Yujie Wu, Siyuan Xu, Jibin Wu, Lei Deng, Mingkun Xu, Qinghao Wen, Guoqi Li","doi":"arxiv-2408.14925","DOIUrl":"https://doi.org/arxiv-2408.14925","url":null,"abstract":"The Forward-Forward (FF) algorithm was recently proposed as a local learning\u0000method to address the limitations of backpropagation (BP), offering biological\u0000plausibility along with memory-efficient and highly parallelized computational\u0000benefits. However, it suffers from suboptimal performance and poor\u0000generalization, largely due to inadequate theoretical support and a lack of\u0000effective learning strategies. In this work, we reformulate FF using distance\u0000metric learning and propose a distance-forward algorithm (DF) to improve FF\u0000performance in supervised vision tasks while preserving its local computational\u0000properties, making it competitive for efficient on-chip learning. To achieve\u0000this, we reinterpret FF through the lens of centroid-based metric learning and\u0000develop a goodness-based N-pair margin loss to facilitate the learning of\u0000discriminative features. Furthermore, we integrate layer-collaboration local\u0000update strategies to reduce information loss caused by greedy local parameter\u0000updates. Our method surpasses existing FF models and other advanced local\u0000learning approaches, with accuracies of 99.7% on MNIST, 88.2% on CIFAR-10,\u000059% on CIFAR-100, 95.9% on SVHN, and 82.5% on ImageNette, respectively.\u0000Moreover, it achieves comparable performance with less than 40% memory cost\u0000compared to BP training, while exhibiting stronger robustness to multiple types\u0000of hardware-related noise, demonstrating its potential for online learning and\u0000energy-efficient computation on neuromorphic chips.","PeriodicalId":501347,"journal":{"name":"arXiv - CS - Neural and Evolutionary Computing","volume":"24 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2024-08-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142188266","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Xinyi Chen, Jibin Wu, Chenxiang Ma, Yinsong Yan, Yujie Wu, Kay Chen Tan
Spiking Neural Networks (SNNs) hold great potential to realize brain-inspired, energy-efficient computational systems. However, current SNNs still fall short in terms of multi-scale temporal processing compared to their biological counterparts. This limitation has resulted in poor performance in many pattern recognition tasks with information that varies across different timescales. To address this issue, we put forward a novel spiking neuron model called Parallel Multi-compartment Spiking Neuron (PMSN). The PMSN emulates biological neurons by incorporating multiple interacting substructures and allows for flexible adjustment of the substructure counts to effectively represent temporal information across diverse timescales. Additionally, to address the computational burden associated with the increased complexity of the proposed model, we introduce two parallelization techniques that decouple the temporal dependencies of neuronal updates, enabling parallelized training across different time steps. Our experimental results on a wide range of pattern recognition tasks demonstrate the superiority of PMSN. It outperforms other state-of-the-art spiking neuron models in terms of its temporal processing capacity, training speed, and computation cost. Specifically, compared with the commonly used Leaky Integrate-and-Fire neuron, PMSN offers a simulation acceleration of over 10 $times$ and a 30 % improvement in accuracy on Sequential CIFAR10 dataset, while maintaining comparable computational cost.
{"title":"PMSN: A Parallel Multi-compartment Spiking Neuron for Multi-scale Temporal Processing","authors":"Xinyi Chen, Jibin Wu, Chenxiang Ma, Yinsong Yan, Yujie Wu, Kay Chen Tan","doi":"arxiv-2408.14917","DOIUrl":"https://doi.org/arxiv-2408.14917","url":null,"abstract":"Spiking Neural Networks (SNNs) hold great potential to realize\u0000brain-inspired, energy-efficient computational systems. However, current SNNs\u0000still fall short in terms of multi-scale temporal processing compared to their\u0000biological counterparts. This limitation has resulted in poor performance in\u0000many pattern recognition tasks with information that varies across different\u0000timescales. To address this issue, we put forward a novel spiking neuron model\u0000called Parallel Multi-compartment Spiking Neuron (PMSN). The PMSN emulates\u0000biological neurons by incorporating multiple interacting substructures and\u0000allows for flexible adjustment of the substructure counts to effectively\u0000represent temporal information across diverse timescales. Additionally, to\u0000address the computational burden associated with the increased complexity of\u0000the proposed model, we introduce two parallelization techniques that decouple\u0000the temporal dependencies of neuronal updates, enabling parallelized training\u0000across different time steps. Our experimental results on a wide range of\u0000pattern recognition tasks demonstrate the superiority of PMSN. It outperforms\u0000other state-of-the-art spiking neuron models in terms of its temporal\u0000processing capacity, training speed, and computation cost. Specifically,\u0000compared with the commonly used Leaky Integrate-and-Fire neuron, PMSN offers a\u0000simulation acceleration of over 10 $times$ and a 30 % improvement in accuracy\u0000on Sequential CIFAR10 dataset, while maintaining comparable computational cost.","PeriodicalId":501347,"journal":{"name":"arXiv - CS - Neural and Evolutionary Computing","volume":"30 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2024-08-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142188267","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Spiking neural networks (SNNs) are gaining popularity in the computational simulation and artificial intelligence fields owing to their biological plausibility and computational efficiency. This paper explores the historical development of SNN and concludes that these two fields are intersecting and merging rapidly. Following the successful application of Dynamic Vision Sensors (DVS) and Dynamic Audio Sensors (DAS), SNNs have found some proper paradigms, such as continuous visual signal tracking, automatic speech recognition, and reinforcement learning for continuous control, that have extensively supported their key features, including spike encoding, neuronal heterogeneity, specific functional circuits, and multiscale plasticity. Compared to these real-world paradigms, the brain contains a spiking version of the biology-world paradigm, which exhibits a similar level of complexity and is usually considered a mirror of the real world. Considering the projected rapid development of invasive and parallel Brain-Computer Interface (BCI), as well as the new BCI-based paradigms that include online pattern recognition and stimulus control of biological spike trains, SNNs naturally leverage their advantages in energy efficiency, robustness, and flexibility. The biological brain has inspired the present study of SNNs and effective SNN machine-learning algorithms, which can help enhance neuroscience discoveries in the brain by applying them to the new BCI paradigm. Such two-way interactions with positive feedback can accelerate brain science research and brain-inspired intelligence technology.
{"title":"Research Advances and New Paradigms for Biology-inspired Spiking Neural Networks","authors":"Tianyu Zheng, Liyuan Han, Tielin Zhang","doi":"arxiv-2408.13996","DOIUrl":"https://doi.org/arxiv-2408.13996","url":null,"abstract":"Spiking neural networks (SNNs) are gaining popularity in the computational\u0000simulation and artificial intelligence fields owing to their biological\u0000plausibility and computational efficiency. This paper explores the historical\u0000development of SNN and concludes that these two fields are intersecting and\u0000merging rapidly. Following the successful application of Dynamic Vision Sensors\u0000(DVS) and Dynamic Audio Sensors (DAS), SNNs have found some proper paradigms,\u0000such as continuous visual signal tracking, automatic speech recognition, and\u0000reinforcement learning for continuous control, that have extensively supported\u0000their key features, including spike encoding, neuronal heterogeneity, specific\u0000functional circuits, and multiscale plasticity. Compared to these real-world\u0000paradigms, the brain contains a spiking version of the biology-world paradigm,\u0000which exhibits a similar level of complexity and is usually considered a mirror\u0000of the real world. Considering the projected rapid development of invasive and\u0000parallel Brain-Computer Interface (BCI), as well as the new BCI-based paradigms\u0000that include online pattern recognition and stimulus control of biological\u0000spike trains, SNNs naturally leverage their advantages in energy efficiency,\u0000robustness, and flexibility. The biological brain has inspired the present\u0000study of SNNs and effective SNN machine-learning algorithms, which can help\u0000enhance neuroscience discoveries in the brain by applying them to the new BCI\u0000paradigm. Such two-way interactions with positive feedback can accelerate brain\u0000science research and brain-inspired intelligence technology.","PeriodicalId":501347,"journal":{"name":"arXiv - CS - Neural and Evolutionary Computing","volume":"8 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2024-08-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142188270","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Uncertainty quantification is an important part of many performance critical applications. This paper provides a simple alternative to existing approaches such as ensemble learning and bayesian neural networks. By directly modeling the loss distribution with an Implicit Quantile Network, we get an estimate of how uncertain the model is of its predictions. For experiments with MNIST and CIFAR datasets, the mean of the estimated loss distribution is 2x higher for incorrect predictions. When data with high estimated uncertainty is removed from the test dataset, the accuracy of the model goes up as much as 10%. This method is simple to implement while offering important information to applications where the user has to know when the model could be wrong (e.g. deep learning for healthcare).
{"title":"Estimating Uncertainty with Implicit Quantile Network","authors":"Yi Hung Lim","doi":"arxiv-2408.14525","DOIUrl":"https://doi.org/arxiv-2408.14525","url":null,"abstract":"Uncertainty quantification is an important part of many performance critical\u0000applications. This paper provides a simple alternative to existing approaches\u0000such as ensemble learning and bayesian neural networks. By directly modeling\u0000the loss distribution with an Implicit Quantile Network, we get an estimate of\u0000how uncertain the model is of its predictions. For experiments with MNIST and\u0000CIFAR datasets, the mean of the estimated loss distribution is 2x higher for\u0000incorrect predictions. When data with high estimated uncertainty is removed\u0000from the test dataset, the accuracy of the model goes up as much as 10%. This\u0000method is simple to implement while offering important information to\u0000applications where the user has to know when the model could be wrong (e.g.\u0000deep learning for healthcare).","PeriodicalId":501347,"journal":{"name":"arXiv - CS - Neural and Evolutionary Computing","volume":"20 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2024-08-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142188269","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pre-trained Artificial Neural Networks (ANNs) exhibit robust pattern recognition capabilities and share extensive similarities with the human brain, specifically Biological Neural Networks (BNNs). We are particularly intrigued by these models' ability to acquire new knowledge through fine-tuning. In this regard, Parameter-efficient Fine-tuning (PEFT) has gained widespread adoption as a substitute for full fine-tuning due to its cost reduction in training and mitigation of over-fitting risks by limiting the number of trainable parameters during adaptation. Since both ANNs and BNNs propagate information layer-by-layer, a common analogy can be drawn: weights in ANNs represent synapses in BNNs, while features (also known as latent variables or logits) in ANNs represent neurotransmitters released by neurons in BNNs. Mainstream PEFT methods aim to adjust feature or parameter values using only a limited number of trainable parameters (usually less than 1% of the total parameters), yet achieve surprisingly good results. Building upon this clue, we delve deeper into exploring the connections between feature adjustment and parameter adjustment, resulting in our proposed method Synapses & Neurons (SAN) that learns scaling matrices for features and propagates their effects towards posterior weight matrices. Our approach draws strong inspiration from well-known neuroscience phenomena - Long-term Potentiation (LTP) and Long-term Depression (LTD), which also reveal the relationship between synapse development and neurotransmitter release levels. We conducted extensive comparisons of PEFT on 26 datasets using attention-based networks as well as convolution-based networks, leading to significant improvements compared to other tuning methods (+8.5% over fully-finetune, +7% over Visual Prompt Tuning, and +3.2% over LoRA). The codes would be released.
{"title":"Discovering Long-Term Effects on Parameter Efficient Fine-tuning","authors":"Gaole Dai, Yiming Tang, Chunkai Fan, Qizhe Zhang, Zhi Zhang, Yulu Gan, Chengqing Zeng, Shanghang Zhang, Tiejun Huang","doi":"arxiv-2409.06706","DOIUrl":"https://doi.org/arxiv-2409.06706","url":null,"abstract":"Pre-trained Artificial Neural Networks (ANNs) exhibit robust pattern\u0000recognition capabilities and share extensive similarities with the human brain,\u0000specifically Biological Neural Networks (BNNs). We are particularly intrigued\u0000by these models' ability to acquire new knowledge through fine-tuning. In this\u0000regard, Parameter-efficient Fine-tuning (PEFT) has gained widespread adoption\u0000as a substitute for full fine-tuning due to its cost reduction in training and\u0000mitigation of over-fitting risks by limiting the number of trainable parameters\u0000during adaptation. Since both ANNs and BNNs propagate information\u0000layer-by-layer, a common analogy can be drawn: weights in ANNs represent\u0000synapses in BNNs, while features (also known as latent variables or logits) in\u0000ANNs represent neurotransmitters released by neurons in BNNs. Mainstream PEFT\u0000methods aim to adjust feature or parameter values using only a limited number\u0000of trainable parameters (usually less than 1% of the total parameters), yet\u0000achieve surprisingly good results. Building upon this clue, we delve deeper\u0000into exploring the connections between feature adjustment and parameter\u0000adjustment, resulting in our proposed method Synapses & Neurons (SAN) that\u0000learns scaling matrices for features and propagates their effects towards\u0000posterior weight matrices. Our approach draws strong inspiration from\u0000well-known neuroscience phenomena - Long-term Potentiation (LTP) and Long-term\u0000Depression (LTD), which also reveal the relationship between synapse\u0000development and neurotransmitter release levels. We conducted extensive\u0000comparisons of PEFT on 26 datasets using attention-based networks as well as\u0000convolution-based networks, leading to significant improvements compared to\u0000other tuning methods (+8.5% over fully-finetune, +7% over Visual Prompt Tuning,\u0000and +3.2% over LoRA). The codes would be released.","PeriodicalId":501347,"journal":{"name":"arXiv - CS - Neural and Evolutionary Computing","volume":"45 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2024-08-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142188274","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Wentao Wu, Fanghua Hong, Xiao Wang, Chenglong Li, Jin Tang
Existing vehicle detectors are usually obtained by training a typical detector (e.g., YOLO, RCNN, DETR series) on vehicle images based on a pre-trained backbone (e.g., ResNet, ViT). Some researchers also exploit and enhance the detection performance using pre-trained large foundation models. However, we think these detectors may only get sub-optimal results because the large models they use are not specifically designed for vehicles. In addition, their results heavily rely on visual features, and seldom of they consider the alignment between the vehicle's semantic information and visual representations. In this work, we propose a new vehicle detection paradigm based on a pre-trained foundation vehicle model (VehicleMAE) and a large language model (T5), termed VFM-Det. It follows the region proposal-based detection framework and the features of each proposal can be enhanced using VehicleMAE. More importantly, we propose a new VAtt2Vec module that predicts the vehicle semantic attributes of these proposals and transforms them into feature vectors to enhance the vision features via contrastive learning. Extensive experiments on three vehicle detection benchmark datasets thoroughly proved the effectiveness of our vehicle detector. Specifically, our model improves the baseline approach by $+5.1%$, $+6.2%$ on the $AP_{0.5}$, $AP_{0.75}$ metrics, respectively, on the Cityscapes dataset.The source code of this work will be released at https://github.com/Event-AHU/VFM-Det.
{"title":"VFM-Det: Towards High-Performance Vehicle Detection via Large Foundation Models","authors":"Wentao Wu, Fanghua Hong, Xiao Wang, Chenglong Li, Jin Tang","doi":"arxiv-2408.13031","DOIUrl":"https://doi.org/arxiv-2408.13031","url":null,"abstract":"Existing vehicle detectors are usually obtained by training a typical\u0000detector (e.g., YOLO, RCNN, DETR series) on vehicle images based on a\u0000pre-trained backbone (e.g., ResNet, ViT). Some researchers also exploit and\u0000enhance the detection performance using pre-trained large foundation models.\u0000However, we think these detectors may only get sub-optimal results because the\u0000large models they use are not specifically designed for vehicles. In addition,\u0000their results heavily rely on visual features, and seldom of they consider the\u0000alignment between the vehicle's semantic information and visual\u0000representations. In this work, we propose a new vehicle detection paradigm\u0000based on a pre-trained foundation vehicle model (VehicleMAE) and a large\u0000language model (T5), termed VFM-Det. It follows the region proposal-based\u0000detection framework and the features of each proposal can be enhanced using\u0000VehicleMAE. More importantly, we propose a new VAtt2Vec module that predicts\u0000the vehicle semantic attributes of these proposals and transforms them into\u0000feature vectors to enhance the vision features via contrastive learning.\u0000Extensive experiments on three vehicle detection benchmark datasets thoroughly\u0000proved the effectiveness of our vehicle detector. Specifically, our model\u0000improves the baseline approach by $+5.1%$, $+6.2%$ on the $AP_{0.5}$,\u0000$AP_{0.75}$ metrics, respectively, on the Cityscapes dataset.The source code of\u0000this work will be released at https://github.com/Event-AHU/VFM-Det.","PeriodicalId":501347,"journal":{"name":"arXiv - CS - Neural and Evolutionary Computing","volume":"51 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2024-08-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142188272","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
The Spiking Neural Network (SNN), due to its unique spiking-driven nature, is a more energy-efficient and effective neural network compared to Artificial Neural Networks (ANNs). The encoding method directly influences the overall performance of the network, and currently, direct encoding is primarily used for directly trained SNNs. When working with static image datasets, direct encoding inputs the same feature map at every time step, failing to fully exploit the spatiotemporal properties of SNNs. While temporal encoding converts input data into spike trains with spatiotemporal characteristics, traditional SNNs utilize the same neurons when processing input data across different time steps, limiting their ability to integrate and utilize spatiotemporal information effectively.To address this, this paper employs temporal encoding and proposes the Adaptive Spiking Neural Network (ASNN), enhancing the utilization of temporal encoding in conventional SNNs. Additionally, temporal encoding is less frequently used because short time steps can lead to significant loss of input data information, often necessitating a higher number of time steps in practical applications. However, training large SNNs with long time steps is challenging due to hardware constraints. To overcome this, this paper introduces a hybrid encoding approach that not only reduces the required time steps for training but also continues to improve the overall network performance.Notably, significant improvements in classification performance are observed on both Spikformer and Spiking ResNet architectures.our code is available at https://github.com/hhx0320/ASNN
{"title":"Adaptive Spiking Neural Networks with Hybrid Coding","authors":"Huaxu He","doi":"arxiv-2408.12407","DOIUrl":"https://doi.org/arxiv-2408.12407","url":null,"abstract":"The Spiking Neural Network (SNN), due to its unique spiking-driven nature, is\u0000a more energy-efficient and effective neural network compared to Artificial\u0000Neural Networks (ANNs). The encoding method directly influences the overall\u0000performance of the network, and currently, direct encoding is primarily used\u0000for directly trained SNNs. When working with static image datasets, direct\u0000encoding inputs the same feature map at every time step, failing to fully\u0000exploit the spatiotemporal properties of SNNs. While temporal encoding converts\u0000input data into spike trains with spatiotemporal characteristics, traditional\u0000SNNs utilize the same neurons when processing input data across different time\u0000steps, limiting their ability to integrate and utilize spatiotemporal\u0000information effectively.To address this, this paper employs temporal encoding\u0000and proposes the Adaptive Spiking Neural Network (ASNN), enhancing the\u0000utilization of temporal encoding in conventional SNNs. Additionally, temporal\u0000encoding is less frequently used because short time steps can lead to\u0000significant loss of input data information, often necessitating a higher number\u0000of time steps in practical applications. However, training large SNNs with long\u0000time steps is challenging due to hardware constraints. To overcome this, this\u0000paper introduces a hybrid encoding approach that not only reduces the required\u0000time steps for training but also continues to improve the overall network\u0000performance.Notably, significant improvements in classification performance are\u0000observed on both Spikformer and Spiking ResNet architectures.our code is\u0000available at https://github.com/hhx0320/ASNN","PeriodicalId":501347,"journal":{"name":"arXiv - CS - Neural and Evolutionary Computing","volume":"7 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2024-08-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142188275","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Abhishek Moitra, Abhiroop Bhattacharjee, Yuhang Li, Youngeun Kim, Priyadarshini Panda
This review explores the intersection of bio-plausible artificial intelligence in the form of Spiking Neural Networks (SNNs) with the analog In-Memory Computing (IMC) domain, highlighting their collective potential for low-power edge computing environments. Through detailed investigation at the device, circuit, and system levels, we highlight the pivotal synergies between SNNs and IMC architectures. Additionally, we emphasize the critical need for comprehensive system-level analyses, considering the inter-dependencies between algorithms, devices, circuit & system parameters, crucial for optimal performance. An in-depth analysis leads to identification of key system-level bottlenecks arising from device limitations which can be addressed using SNN-specific algorithm-hardware co-design techniques. This review underscores the imperative for holistic device to system design space co-exploration, highlighting the critical aspects of hardware and algorithm research endeavors for low-power neuromorphic solutions.
{"title":"When In-memory Computing Meets Spiking Neural Networks -- A Perspective on Device-Circuit-System-and-Algorithm Co-design","authors":"Abhishek Moitra, Abhiroop Bhattacharjee, Yuhang Li, Youngeun Kim, Priyadarshini Panda","doi":"arxiv-2408.12767","DOIUrl":"https://doi.org/arxiv-2408.12767","url":null,"abstract":"This review explores the intersection of bio-plausible artificial\u0000intelligence in the form of Spiking Neural Networks (SNNs) with the analog\u0000In-Memory Computing (IMC) domain, highlighting their collective potential for\u0000low-power edge computing environments. Through detailed investigation at the\u0000device, circuit, and system levels, we highlight the pivotal synergies between\u0000SNNs and IMC architectures. Additionally, we emphasize the critical need for\u0000comprehensive system-level analyses, considering the inter-dependencies between\u0000algorithms, devices, circuit & system parameters, crucial for optimal\u0000performance. An in-depth analysis leads to identification of key system-level\u0000bottlenecks arising from device limitations which can be addressed using\u0000SNN-specific algorithm-hardware co-design techniques. This review underscores\u0000the imperative for holistic device to system design space co-exploration,\u0000highlighting the critical aspects of hardware and algorithm research endeavors\u0000for low-power neuromorphic solutions.","PeriodicalId":501347,"journal":{"name":"arXiv - CS - Neural and Evolutionary Computing","volume":"38 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2024-08-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142188271","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Recently, AI research has primarily focused on large language models (LLMs), and increasing accuracy often involves scaling up and consuming more power. The power consumption of AI has become a significant societal issue; in this context, spiking neural networks (SNNs) offer a promising solution. SNNs operate event-driven, like the human brain, and compress information temporally. These characteristics allow SNNs to significantly reduce power consumption compared to perceptron-based artificial neural networks (ANNs), highlighting them as a next-generation neural network technology. However, societal concerns regarding AI go beyond power consumption, with the reliability of AI models being a global issue. For instance, adversarial attacks on AI models are a well-studied problem in the context of traditional neural networks. Despite their importance, the stability and property verification of SNNs remains in the early stages of research. Most SNN verification methods are time-consuming and barely scalable, making practical applications challenging. In this paper, we introduce temporal encoding to achieve practical performance in verifying the adversarial robustness of SNNs. We conduct a theoretical analysis of this approach and demonstrate its success in verifying SNNs at previously unmanageable scales. Our contribution advances SNN verification to a practical level, facilitating the safer application of SNNs.
{"title":"Towards Efficient Formal Verification of Spiking Neural Network","authors":"Baekryun Seong, Jieung Kim, Sang-Ki Ko","doi":"arxiv-2408.10900","DOIUrl":"https://doi.org/arxiv-2408.10900","url":null,"abstract":"Recently, AI research has primarily focused on large language models (LLMs),\u0000and increasing accuracy often involves scaling up and consuming more power. The\u0000power consumption of AI has become a significant societal issue; in this\u0000context, spiking neural networks (SNNs) offer a promising solution. SNNs\u0000operate event-driven, like the human brain, and compress information\u0000temporally. These characteristics allow SNNs to significantly reduce power\u0000consumption compared to perceptron-based artificial neural networks (ANNs),\u0000highlighting them as a next-generation neural network technology. However,\u0000societal concerns regarding AI go beyond power consumption, with the\u0000reliability of AI models being a global issue. For instance, adversarial\u0000attacks on AI models are a well-studied problem in the context of traditional\u0000neural networks. Despite their importance, the stability and property\u0000verification of SNNs remains in the early stages of research. Most SNN\u0000verification methods are time-consuming and barely scalable, making practical\u0000applications challenging. In this paper, we introduce temporal encoding to\u0000achieve practical performance in verifying the adversarial robustness of SNNs.\u0000We conduct a theoretical analysis of this approach and demonstrate its success\u0000in verifying SNNs at previously unmanageable scales. Our contribution advances\u0000SNN verification to a practical level, facilitating the safer application of\u0000SNNs.","PeriodicalId":501347,"journal":{"name":"arXiv - CS - Neural and Evolutionary Computing","volume":"30 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2024-08-20","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142188308","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Ruiyuan Kang, Panos Liatsis, Meixia Geng, Qingjie Yang
Laser absorption spectroscopy (LAS) quantification is a popular tool used in measuring temperature and concentration of gases. It has low error tolerance, whereas current ML-based solutions cannot guarantee their measure reliability. In this work, we propose a new framework, SPEC, to address this issue. In addition to the conventional ML estimator-based estimation mode, SPEC also includes a Physics-driven Anomaly Detection module (PAD) to assess the error of the estimation. And a Correction mode is designed to correct the unreliable estimation. The correction mode is a network-based optimization algorithm, which uses the guidance of error to iteratively correct the estimation. A hybrid surrogate error model is proposed to estimate the error distribution, which contains an ensemble of networks to simulate reconstruction error, and true feasible error computation. A greedy ensemble search is proposed to find the optimal correction robustly and efficiently from the gradient guidance of surrogate model. The proposed SPEC is validated on the test scenarios which are outside the training distribution. The results show that SPEC can significantly improve the estimation quality, and the correction mode outperforms current network-based optimization algorithms. In addition, SPEC has the reconfigurability, which can be easily adapted to different quantification tasks via changing PAD without retraining the ML estimator.
激光吸收光谱（LAS）定量是测量温度和气体浓度的常用工具。在这项工作中，我们提出了一个新的框架 SPEC 来解决这个问题。除了传统的基于 ML 估算器的估算模式外，SPEC 还包括一个物理驱动的异常检测模块（PAD），用于评估估算误差。此外，还设计了一种修正模式来纠正不可靠的估计。修正模式是一种基于网络的优化算法，它利用误差的指导来迭代修正估算。提出了一种混合代用误差模型来估计误差分布，该模型包含模拟重建误差的网络集合和真实可行误差计算。提出了一种贪婪集合搜索方法，以便从代理模型的梯度引导中稳健高效地找到最优修正。提出的 SPEC 在训练分布之外的测试场景中进行了验证。结果表明，SPEC 可以显著提高估计质量，其修正模式优于当前基于网络的优化算法。此外，SPEC 还具有可配置性，可以通过改变 PAD 轻松适应不同的量化任务，而无需重新训练 ML 估计器。
{"title":"Physics-Driven AI Correction in Laser Absorption Sensing Quantification","authors":"Ruiyuan Kang, Panos Liatsis, Meixia Geng, Qingjie Yang","doi":"arxiv-2408.10714","DOIUrl":"https://doi.org/arxiv-2408.10714","url":null,"abstract":"Laser absorption spectroscopy (LAS) quantification is a popular tool used in\u0000measuring temperature and concentration of gases. It has low error tolerance,\u0000whereas current ML-based solutions cannot guarantee their measure reliability.\u0000In this work, we propose a new framework, SPEC, to address this issue. In\u0000addition to the conventional ML estimator-based estimation mode, SPEC also\u0000includes a Physics-driven Anomaly Detection module (PAD) to assess the error of\u0000the estimation. And a Correction mode is designed to correct the unreliable\u0000estimation. The correction mode is a network-based optimization algorithm,\u0000which uses the guidance of error to iteratively correct the estimation. A\u0000hybrid surrogate error model is proposed to estimate the error distribution,\u0000which contains an ensemble of networks to simulate reconstruction error, and\u0000true feasible error computation. A greedy ensemble search is proposed to find\u0000the optimal correction robustly and efficiently from the gradient guidance of\u0000surrogate model. The proposed SPEC is validated on the test scenarios which are\u0000outside the training distribution. The results show that SPEC can significantly\u0000improve the estimation quality, and the correction mode outperforms current\u0000network-based optimization algorithms. In addition, SPEC has the\u0000reconfigurability, which can be easily adapted to different quantification\u0000tasks via changing PAD without retraining the ML estimator.","PeriodicalId":501347,"journal":{"name":"arXiv - CS - Neural and Evolutionary Computing","volume":"76 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2024-08-20","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142188277","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：