Oscar Dilley, Juan Marcelo Parra-Ullauri, Rasheed Hussain, Dimitra Simeonidou
Federated Learning (FL) is a privacy-enhancing technology for distributed ML. By training models locally and aggregating updates - a federation learns together, while bypassing centralised data collection. FL is increasingly popular in healthcare, finance and personal computing. However, it inherits fairness challenges from classical ML and introduces new ones, resulting from differences in data quality, client participation, communication constraints, aggregation methods and underlying hardware. Fairness remains an unresolved issue in FL and the community has identified an absence of succinct definitions and metrics to quantify fairness; to address this, we propose Federated Fairness Analytics - a methodology for measuring fairness. Our definition of fairness comprises four notions with novel, corresponding metrics. They are symptomatically defined and leverage techniques originating from XAI, cooperative game-theory and networking engineering. We tested a range of experimental settings, varying the FL approach, ML task and data settings. The results show that statistical heterogeneity and client participation affect fairness and fairness conscious approaches such as Ditto and q-FedAvg marginally improve fairness-performance trade-offs. Using our techniques, FL practitioners can uncover previously unobtainable insights into their system's fairness, at differing levels of granularity in order to address fairness challenges in FL. We have open-sourced our work at: https://github.com/oscardilley/federated-fairness.
{"title":"Federated Fairness Analytics: Quantifying Fairness in Federated Learning","authors":"Oscar Dilley, Juan Marcelo Parra-Ullauri, Rasheed Hussain, Dimitra Simeonidou","doi":"arxiv-2408.08214","DOIUrl":"https://doi.org/arxiv-2408.08214","url":null,"abstract":"Federated Learning (FL) is a privacy-enhancing technology for distributed ML.\u0000By training models locally and aggregating updates - a federation learns\u0000together, while bypassing centralised data collection. FL is increasingly\u0000popular in healthcare, finance and personal computing. However, it inherits\u0000fairness challenges from classical ML and introduces new ones, resulting from\u0000differences in data quality, client participation, communication constraints,\u0000aggregation methods and underlying hardware. Fairness remains an unresolved\u0000issue in FL and the community has identified an absence of succinct definitions\u0000and metrics to quantify fairness; to address this, we propose Federated\u0000Fairness Analytics - a methodology for measuring fairness. Our definition of\u0000fairness comprises four notions with novel, corresponding metrics. They are\u0000symptomatically defined and leverage techniques originating from XAI,\u0000cooperative game-theory and networking engineering. We tested a range of\u0000experimental settings, varying the FL approach, ML task and data settings. The\u0000results show that statistical heterogeneity and client participation affect\u0000fairness and fairness conscious approaches such as Ditto and q-FedAvg\u0000marginally improve fairness-performance trade-offs. Using our techniques, FL\u0000practitioners can uncover previously unobtainable insights into their system's\u0000fairness, at differing levels of granularity in order to address fairness\u0000challenges in FL. We have open-sourced our work at:\u0000https://github.com/oscardilley/federated-fairness.","PeriodicalId":501347,"journal":{"name":"arXiv - CS - Neural and Evolutionary Computing","volume":"14 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2024-08-15","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142188307","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Reservoir Computing (RC) models, a subclass of recurrent neural networks, are distinguished by their fixed, non-trainable input layer and dynamically coupled reservoir, with only the static readout layer being trained. This design circumvents the issues associated with backpropagating error signals through time, thereby enhancing both stability and training efficiency. RC models have been successfully applied across a broad range of application domains. Crucially, they have been demonstrated to be universal approximators of time-invariant dynamic filters with fading memory, under various settings of approximation norms and input driving sources. Simple Cycle Reservoirs (SCR) represent a specialized class of RC models with a highly constrained reservoir architecture, characterized by uniform ring connectivity and binary input-to-reservoir weights with an aperiodic sign pattern. For linear reservoirs, given the reservoir size, the reservoir construction has only one degree of freedom -- the reservoir cycle weight. Such architectures are particularly amenable to hardware implementations without significant performance degradation in many practical tasks. In this study we endow these observations with solid theoretical foundations by proving that SCRs operating in real domain are universal approximators of time-invariant dynamic filters with fading memory. Our results supplement recent research showing that SCRs in the complex domain can approximate, to arbitrary precision, any unrestricted linear reservoir with a non-linear readout. We furthermore introduce a novel method to drastically reduce the number of SCR units, making such highly constrained architectures natural candidates for low-complexity hardware implementations. Our findings are supported by empirical studies on real-world time series datasets.
{"title":"Universality of Real Minimal Complexity Reservoir","authors":"Robert Simon Fong, Boyu Li, Peter Tiňo","doi":"arxiv-2408.08071","DOIUrl":"https://doi.org/arxiv-2408.08071","url":null,"abstract":"Reservoir Computing (RC) models, a subclass of recurrent neural networks, are\u0000distinguished by their fixed, non-trainable input layer and dynamically coupled\u0000reservoir, with only the static readout layer being trained. This design\u0000circumvents the issues associated with backpropagating error signals through\u0000time, thereby enhancing both stability and training efficiency. RC models have\u0000been successfully applied across a broad range of application domains.\u0000Crucially, they have been demonstrated to be universal approximators of\u0000time-invariant dynamic filters with fading memory, under various settings of\u0000approximation norms and input driving sources. Simple Cycle Reservoirs (SCR) represent a specialized class of RC models with\u0000a highly constrained reservoir architecture, characterized by uniform ring\u0000connectivity and binary input-to-reservoir weights with an aperiodic sign\u0000pattern. For linear reservoirs, given the reservoir size, the reservoir\u0000construction has only one degree of freedom -- the reservoir cycle weight. Such\u0000architectures are particularly amenable to hardware implementations without\u0000significant performance degradation in many practical tasks. In this study we\u0000endow these observations with solid theoretical foundations by proving that\u0000SCRs operating in real domain are universal approximators of time-invariant\u0000dynamic filters with fading memory. Our results supplement recent research\u0000showing that SCRs in the complex domain can approximate, to arbitrary\u0000precision, any unrestricted linear reservoir with a non-linear readout. We\u0000furthermore introduce a novel method to drastically reduce the number of SCR\u0000units, making such highly constrained architectures natural candidates for\u0000low-complexity hardware implementations. Our findings are supported by\u0000empirical studies on real-world time series datasets.","PeriodicalId":501347,"journal":{"name":"arXiv - CS - Neural and Evolutionary Computing","volume":"87 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2024-08-15","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142188309","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Maximilian Baronig, Romain Ferrand, Silvester Sabathiel, Robert Legenstein
Efficient implementations of spiking neural networks on neuromorphic hardware promise orders of magnitude less power consumption than their non-spiking counterparts. The standard neuron model for spike-based computation on such neuromorphic systems has long been the leaky integrate-and-fire (LIF) neuron. As a promising advancement, a computationally light augmentation of the LIF neuron model with an adaptation mechanism experienced a recent upswing in popularity, caused by demonstrations of its superior performance on spatio-temporal processing tasks. The root of the superiority of these so-called adaptive LIF neurons however, is not well understood. In this article, we thoroughly analyze the dynamical, computational, and learning properties of adaptive LIF neurons and networks thereof. We find that the frequently observed stability problems during training of such networks can be overcome by applying an alternative discretization method that results in provably better stability properties than the commonly used Euler-Forward method. With this discretization, we achieved a new state-of-the-art performance on common event-based benchmark datasets. We also show that the superiority of networks of adaptive LIF neurons extends to the prediction and generation of complex time series. Our further analysis of the computational properties of networks of adaptive LIF neurons shows that they are particularly well suited to exploit the spatio-temporal structure of input sequences. Furthermore, these networks are surprisingly robust to shifts of the mean input strength and input spike rate, even when these shifts were not observed during training. As a consequence, high-performance networks can be obtained without any normalization techniques such as batch normalization or batch-normalization through time.
{"title":"Advancing Spatio-Temporal Processing in Spiking Neural Networks through Adaptation","authors":"Maximilian Baronig, Romain Ferrand, Silvester Sabathiel, Robert Legenstein","doi":"arxiv-2408.07517","DOIUrl":"https://doi.org/arxiv-2408.07517","url":null,"abstract":"Efficient implementations of spiking neural networks on neuromorphic hardware\u0000promise orders of magnitude less power consumption than their non-spiking\u0000counterparts. The standard neuron model for spike-based computation on such\u0000neuromorphic systems has long been the leaky integrate-and-fire (LIF) neuron.\u0000As a promising advancement, a computationally light augmentation of the LIF\u0000neuron model with an adaptation mechanism experienced a recent upswing in\u0000popularity, caused by demonstrations of its superior performance on\u0000spatio-temporal processing tasks. The root of the superiority of these\u0000so-called adaptive LIF neurons however, is not well understood. In this\u0000article, we thoroughly analyze the dynamical, computational, and learning\u0000properties of adaptive LIF neurons and networks thereof. We find that the\u0000frequently observed stability problems during training of such networks can be\u0000overcome by applying an alternative discretization method that results in\u0000provably better stability properties than the commonly used Euler-Forward\u0000method. With this discretization, we achieved a new state-of-the-art\u0000performance on common event-based benchmark datasets. We also show that the\u0000superiority of networks of adaptive LIF neurons extends to the prediction and\u0000generation of complex time series. Our further analysis of the computational\u0000properties of networks of adaptive LIF neurons shows that they are particularly\u0000well suited to exploit the spatio-temporal structure of input sequences.\u0000Furthermore, these networks are surprisingly robust to shifts of the mean input\u0000strength and input spike rate, even when these shifts were not observed during\u0000training. As a consequence, high-performance networks can be obtained without\u0000any normalization techniques such as batch normalization or batch-normalization\u0000through time.","PeriodicalId":501347,"journal":{"name":"arXiv - CS - Neural and Evolutionary Computing","volume":"44 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2024-08-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142188310","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Xiaoming Xue, Yao Hu, Liang Feng, Kai Zhang, Linqi Song, Kay Chen Tan
Expensive optimization problems (EOPs) have attracted increasing research attention over the decades due to their ubiquity in a variety of practical applications. Despite many sophisticated surrogate-assisted evolutionary algorithms (SAEAs) that have been developed for solving such problems, most of them lack the ability to transfer knowledge from previously-solved tasks and always start their search from scratch, making them troubled by the notorious cold-start issue. A few preliminary studies that integrate transfer learning into SAEAs still face some issues, such as defective similarity quantification that is prone to underestimate promising knowledge, surrogate-dependency that makes the transfer methods not coherent with the state-of-the-art in SAEAs, etc. In light of the above, a plug and play competitive knowledge transfer method is proposed to boost various SAEAs in this paper. Specifically, both the optimized solutions from the source tasks and the promising solutions acquired by the target surrogate are treated as task-solving knowledge, enabling them to compete with each other to elect the winner for expensive evaluation, thus boosting the search speed on the target task. Moreover, the lower bound of the convergence gain brought by the knowledge competition is mathematically analyzed, which is expected to strengthen the theoretical foundation of sequential transfer optimization. Experimental studies conducted on a series of benchmark problems and a practical application from the petroleum industry verify the efficacy of the proposed method. The source code of the competitive knowledge transfer is available at https://github.com/XmingHsueh/SAS-CKT.
{"title":"Surrogate-Assisted Search with Competitive Knowledge Transfer for Expensive Optimization","authors":"Xiaoming Xue, Yao Hu, Liang Feng, Kai Zhang, Linqi Song, Kay Chen Tan","doi":"arxiv-2408.07176","DOIUrl":"https://doi.org/arxiv-2408.07176","url":null,"abstract":"Expensive optimization problems (EOPs) have attracted increasing research\u0000attention over the decades due to their ubiquity in a variety of practical\u0000applications. Despite many sophisticated surrogate-assisted evolutionary\u0000algorithms (SAEAs) that have been developed for solving such problems, most of\u0000them lack the ability to transfer knowledge from previously-solved tasks and\u0000always start their search from scratch, making them troubled by the notorious\u0000cold-start issue. A few preliminary studies that integrate transfer learning\u0000into SAEAs still face some issues, such as defective similarity quantification\u0000that is prone to underestimate promising knowledge, surrogate-dependency that\u0000makes the transfer methods not coherent with the state-of-the-art in SAEAs,\u0000etc. In light of the above, a plug and play competitive knowledge transfer\u0000method is proposed to boost various SAEAs in this paper. Specifically, both the\u0000optimized solutions from the source tasks and the promising solutions acquired\u0000by the target surrogate are treated as task-solving knowledge, enabling them to\u0000compete with each other to elect the winner for expensive evaluation, thus\u0000boosting the search speed on the target task. Moreover, the lower bound of the\u0000convergence gain brought by the knowledge competition is mathematically\u0000analyzed, which is expected to strengthen the theoretical foundation of\u0000sequential transfer optimization. Experimental studies conducted on a series of\u0000benchmark problems and a practical application from the petroleum industry\u0000verify the efficacy of the proposed method. The source code of the competitive\u0000knowledge transfer is available at https://github.com/XmingHsueh/SAS-CKT.","PeriodicalId":501347,"journal":{"name":"arXiv - CS - Neural and Evolutionary Computing","volume":"23 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2024-08-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142188177","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Ensuring energy-efficient design in neuromorphic computing systems necessitates a tailored architecture combined with algorithmic approaches. This manuscript focuses on enhancing brain-inspired perceptual computing machines through a novel combined learning approach for Convolutional Spiking Neural Networks (CSNNs). CSNNs present a promising alternative to traditional power-intensive and complex machine learning methods like backpropagation, offering energy-efficient spiking neuron processing inspired by the human brain. The proposed combined learning method integrates Pair-based Spike Timing-Dependent Plasticity (PSTDP) and power law-dependent Spike-timing-dependent plasticity (STDP) to adjust synaptic efficacies, enabling the utilization of stochastic elements like memristive devices to enhance energy efficiency and improve perceptual computing accuracy. By reducing learning parameters while maintaining accuracy, these systems consume less energy and have reduced area overhead, making them more suitable for hardware implementation. The research delves into neuromorphic design architectures, focusing on CSNNs to provide a general framework for energy-efficient computing hardware. Various CSNN architectures are evaluated to assess how less trainable parameters can maintain acceptable accuracy in perceptual computing systems, positioning them as viable candidates for neuromorphic architecture. Comparisons with previous work validate the achievements and methodology of the proposed architecture.
{"title":"The Potential of Combined Learning Strategies to Enhance Energy Efficiency of Spiking Neuromorphic Systems","authors":"Ali Shiri Sichani, Sai Kankatala","doi":"arxiv-2408.07150","DOIUrl":"https://doi.org/arxiv-2408.07150","url":null,"abstract":"Ensuring energy-efficient design in neuromorphic computing systems\u0000necessitates a tailored architecture combined with algorithmic approaches. This\u0000manuscript focuses on enhancing brain-inspired perceptual computing machines\u0000through a novel combined learning approach for Convolutional Spiking Neural\u0000Networks (CSNNs). CSNNs present a promising alternative to traditional\u0000power-intensive and complex machine learning methods like backpropagation,\u0000offering energy-efficient spiking neuron processing inspired by the human\u0000brain. The proposed combined learning method integrates Pair-based Spike\u0000Timing-Dependent Plasticity (PSTDP) and power law-dependent\u0000Spike-timing-dependent plasticity (STDP) to adjust synaptic efficacies,\u0000enabling the utilization of stochastic elements like memristive devices to\u0000enhance energy efficiency and improve perceptual computing accuracy. By\u0000reducing learning parameters while maintaining accuracy, these systems consume\u0000less energy and have reduced area overhead, making them more suitable for\u0000hardware implementation. The research delves into neuromorphic design\u0000architectures, focusing on CSNNs to provide a general framework for\u0000energy-efficient computing hardware. Various CSNN architectures are evaluated\u0000to assess how less trainable parameters can maintain acceptable accuracy in\u0000perceptual computing systems, positioning them as viable candidates for\u0000neuromorphic architecture. Comparisons with previous work validate the\u0000achievements and methodology of the proposed architecture.","PeriodicalId":501347,"journal":{"name":"arXiv - CS - Neural and Evolutionary Computing","volume":"28 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2024-08-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142188364","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Rasa Khosrowshahli, Shahryar Rahnamayan, Beatrice Ombuki-Berman
Deep learning is mainly based on utilizing gradient-based optimization for training Deep Neural Network (DNN) models. Although robust and widely used, gradient-based optimization algorithms are prone to getting stuck in local minima. In this modern deep learning era, the state-of-the-art DNN models have millions and billions of parameters, including weights and biases, making them huge-scale optimization problems in terms of search space. Tuning a huge number of parameters is a challenging task that causes vanishing/exploding gradients and overfitting; likewise, utilized loss functions do not exactly represent our targeted performance metrics. A practical solution to exploring large and complex solution space is meta-heuristic algorithms. Since DNNs exceed thousands and millions of parameters, even robust meta-heuristic algorithms, such as Differential Evolution, struggle to efficiently explore and converge in such huge-dimensional search spaces, leading to very slow convergence and high memory demand. To tackle the mentioned curse of dimensionality, the concept of blocking was recently proposed as a technique that reduces the search space dimensions by grouping them into blocks. In this study, we aim to introduce Histogram-based Blocking Differential Evolution (HBDE), a novel approach that hybridizes gradient-based and gradient-free algorithms to optimize parameters. Experimental results demonstrated that the HBDE could reduce the parameters in the ResNet-18 model from 11M to 3K during the training/optimizing phase by metaheuristics, namely, the proposed HBDE, which outperforms baseline gradient-based and parent gradient-free DE algorithms evaluated on CIFAR-10 and CIFAR-100 datasets showcasing its effectiveness with reduced computational demands for the very first time.
{"title":"Massive Dimensions Reduction and Hybridization with Meta-heuristics in Deep Learning","authors":"Rasa Khosrowshahli, Shahryar Rahnamayan, Beatrice Ombuki-Berman","doi":"arxiv-2408.07194","DOIUrl":"https://doi.org/arxiv-2408.07194","url":null,"abstract":"Deep learning is mainly based on utilizing gradient-based optimization for\u0000training Deep Neural Network (DNN) models. Although robust and widely used,\u0000gradient-based optimization algorithms are prone to getting stuck in local\u0000minima. In this modern deep learning era, the state-of-the-art DNN models have\u0000millions and billions of parameters, including weights and biases, making them\u0000huge-scale optimization problems in terms of search space. Tuning a huge number\u0000of parameters is a challenging task that causes vanishing/exploding gradients\u0000and overfitting; likewise, utilized loss functions do not exactly represent our\u0000targeted performance metrics. A practical solution to exploring large and\u0000complex solution space is meta-heuristic algorithms. Since DNNs exceed\u0000thousands and millions of parameters, even robust meta-heuristic algorithms,\u0000such as Differential Evolution, struggle to efficiently explore and converge in\u0000such huge-dimensional search spaces, leading to very slow convergence and high\u0000memory demand. To tackle the mentioned curse of dimensionality, the concept of\u0000blocking was recently proposed as a technique that reduces the search space\u0000dimensions by grouping them into blocks. In this study, we aim to introduce\u0000Histogram-based Blocking Differential Evolution (HBDE), a novel approach that\u0000hybridizes gradient-based and gradient-free algorithms to optimize parameters.\u0000Experimental results demonstrated that the HBDE could reduce the parameters in\u0000the ResNet-18 model from 11M to 3K during the training/optimizing phase by\u0000metaheuristics, namely, the proposed HBDE, which outperforms baseline\u0000gradient-based and parent gradient-free DE algorithms evaluated on CIFAR-10 and\u0000CIFAR-100 datasets showcasing its effectiveness with reduced computational\u0000demands for the very first time.","PeriodicalId":501347,"journal":{"name":"arXiv - CS - Neural and Evolutionary Computing","volume":"18 1 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2024-08-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142188362","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Currently, neural-network processing in machine learning applications relies on layer synchronization, whereby neurons in a layer aggregate incoming currents from all neurons in the preceding layer, before evaluating their activation function. This is practiced even in artificial Spiking Neural Networks (SNNs), which are touted as consistent with neurobiology, in spite of processing in the brain being, in fact asynchronous. A truly asynchronous system however would allow all neurons to evaluate concurrently their threshold and emit spikes upon receiving any presynaptic current. Omitting layer synchronization is potentially beneficial, for latency and energy efficiency, but asynchronous execution of models previously trained with layer synchronization may entail a mismatch in network dynamics and performance. We present a study that documents and quantifies this problem in three datasets on our simulation environment that implements network asynchrony, and we show that models trained with layer synchronization either perform sub-optimally in absence of the synchronization, or they will fail to benefit from any energy and latency reduction, when such a mechanism is in place. We then "make ends meet" and address the problem with unlayered backprop, a novel backpropagation-based training method, for learning models suitable for asynchronous processing. We train with it models that use different neuron execution scheduling strategies, and we show that although their neurons are more reactive, these models consistently exhibit lower overall spike density (up to 50%), reach a correct decision faster (up to 2x) without integrating all spikes, and achieve superior accuracy (up to 10% higher). Our findings suggest that asynchronous event-based (neuromorphic) AI computing is indeed more efficient, but we need to seriously rethink how we train our SNN models, to benefit from it.
{"title":"Overcoming the Limitations of Layer Synchronization in Spiking Neural Networks","authors":"Roel Koopman, Amirreza Yousefzadeh, Mahyar Shahsavari, Guangzhi Tang, Manolis Sifalakis","doi":"arxiv-2408.05098","DOIUrl":"https://doi.org/arxiv-2408.05098","url":null,"abstract":"Currently, neural-network processing in machine learning applications relies\u0000on layer synchronization, whereby neurons in a layer aggregate incoming\u0000currents from all neurons in the preceding layer, before evaluating their\u0000activation function. This is practiced even in artificial Spiking Neural\u0000Networks (SNNs), which are touted as consistent with neurobiology, in spite of\u0000processing in the brain being, in fact asynchronous. A truly asynchronous\u0000system however would allow all neurons to evaluate concurrently their threshold\u0000and emit spikes upon receiving any presynaptic current. Omitting layer\u0000synchronization is potentially beneficial, for latency and energy efficiency,\u0000but asynchronous execution of models previously trained with layer\u0000synchronization may entail a mismatch in network dynamics and performance. We\u0000present a study that documents and quantifies this problem in three datasets on\u0000our simulation environment that implements network asynchrony, and we show that\u0000models trained with layer synchronization either perform sub-optimally in\u0000absence of the synchronization, or they will fail to benefit from any energy\u0000and latency reduction, when such a mechanism is in place. We then \"make ends\u0000meet\" and address the problem with unlayered backprop, a novel\u0000backpropagation-based training method, for learning models suitable for\u0000asynchronous processing. We train with it models that use different neuron\u0000execution scheduling strategies, and we show that although their neurons are\u0000more reactive, these models consistently exhibit lower overall spike density\u0000(up to 50%), reach a correct decision faster (up to 2x) without integrating all\u0000spikes, and achieve superior accuracy (up to 10% higher). Our findings suggest\u0000that asynchronous event-based (neuromorphic) AI computing is indeed more\u0000efficient, but we need to seriously rethink how we train our SNN models, to\u0000benefit from it.","PeriodicalId":501347,"journal":{"name":"arXiv - CS - Neural and Evolutionary Computing","volume":"93 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2024-08-09","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141945505","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
The Keyword Spotting (KWS) task involves continuous audio stream monitoring to detect predefined words, requiring low energy devices for continuous processing. Neuromorphic devices effectively address this energy challenge. However, the general neuromorphic KWS pipeline, from microphone to Spiking Neural Network (SNN), entails multiple processing stages. Leveraging the popularity of Pulse Density Modulation (PDM) microphones in modern devices and their similarity to spiking neurons, we propose a direct microphone-to-SNN connection. This approach eliminates intermediate stages, notably reducing computational costs. The system achieved an accuracy of 91.54% on the Google Speech Command (GSC) dataset, surpassing the state-of-the-art for the Spiking Speech Command (SSC) dataset which is a bio-inspired encoded GSC. Furthermore, the observed sparsity in network activity and connectivity indicates potential for remarkably low energy consumption in a neuromorphic device implementation.
{"title":"Neuromorphic Keyword Spotting with Pulse Density Modulation MEMS Microphones","authors":"Sidi Yaya Arnaud Yarga, Sean U. N. Wood","doi":"arxiv-2408.05156","DOIUrl":"https://doi.org/arxiv-2408.05156","url":null,"abstract":"The Keyword Spotting (KWS) task involves continuous audio stream monitoring\u0000to detect predefined words, requiring low energy devices for continuous\u0000processing. Neuromorphic devices effectively address this energy challenge.\u0000However, the general neuromorphic KWS pipeline, from microphone to Spiking\u0000Neural Network (SNN), entails multiple processing stages. Leveraging the\u0000popularity of Pulse Density Modulation (PDM) microphones in modern devices and\u0000their similarity to spiking neurons, we propose a direct microphone-to-SNN\u0000connection. This approach eliminates intermediate stages, notably reducing\u0000computational costs. The system achieved an accuracy of 91.54% on the Google\u0000Speech Command (GSC) dataset, surpassing the state-of-the-art for the Spiking\u0000Speech Command (SSC) dataset which is a bio-inspired encoded GSC. Furthermore,\u0000the observed sparsity in network activity and connectivity indicates potential\u0000for remarkably low energy consumption in a neuromorphic device implementation.","PeriodicalId":501347,"journal":{"name":"arXiv - CS - Neural and Evolutionary Computing","volume":"26 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2024-08-09","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141945504","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Javier Hernández-Tello, Miguel Ángel Martínez-del-Amor, David Orellana-Martín, Francis George C. Cabarle
The parallel simulation of Spiking Neural P systems is mainly based on a matrix representation, where the graph inherent to the neural model is encoded in an adjacency matrix. The simulation algorithm is based on a matrix-vector multiplication, which is an operation efficiently implemented on parallel devices. However, when the graph of a Spiking Neural P system is not fully connected, the adjacency matrix is sparse and hence, lots of computing resources are wasted in both time and memory domains. For this reason, two compression methods for the matrix representation were proposed in a previous work, but they were not implemented nor parallelized on a simulator. In this paper, they are implemented and parallelized on GPUs as part of a new Spiking Neural P system with delays simulator. Extensive experiments are conducted on high-end GPUs (RTX2080 and A100 80GB), and it is concluded that they outperform other solutions based on state-of-the-art GPU libraries when simulating Spiking Neural P systems.
尖峰神经 P 系统的并行仿真主要基于矩阵表示法，其中神经模型的固有图被编码为邻接矩阵。仿真算法基于矩阵向量相乘，这是在并行设备上高效实现的操作。然而，当尖峰神经 P 系统的图不是完全连接时，邻接矩阵是稀疏的，因此在时间和内存领域都会浪费大量计算资源。为此，前人提出了矩阵表示的两种压缩方法，但没有在模拟器上实现或并行化。本文在 GPU 上实现了这两种方法，并将其并行化，作为带有延迟模拟器的新型 SpikingNeural P 系统的一部分。在高端 GPU（RTX2080 和 A100 80GB）上进行了广泛的实验，结论是在模拟尖峰神经 P 系统时，它们优于基于最先进 GPU 库的其他解决方案。
{"title":"Sparse Spiking Neural-like Membrane Systems on Graphics Processing Units","authors":"Javier Hernández-Tello, Miguel Ángel Martínez-del-Amor, David Orellana-Martín, Francis George C. Cabarle","doi":"arxiv-2408.04343","DOIUrl":"https://doi.org/arxiv-2408.04343","url":null,"abstract":"The parallel simulation of Spiking Neural P systems is mainly based on a\u0000matrix representation, where the graph inherent to the neural model is encoded\u0000in an adjacency matrix. The simulation algorithm is based on a matrix-vector\u0000multiplication, which is an operation efficiently implemented on parallel\u0000devices. However, when the graph of a Spiking Neural P system is not fully\u0000connected, the adjacency matrix is sparse and hence, lots of computing\u0000resources are wasted in both time and memory domains. For this reason, two\u0000compression methods for the matrix representation were proposed in a previous\u0000work, but they were not implemented nor parallelized on a simulator. In this\u0000paper, they are implemented and parallelized on GPUs as part of a new Spiking\u0000Neural P system with delays simulator. Extensive experiments are conducted on\u0000high-end GPUs (RTX2080 and A100 80GB), and it is concluded that they outperform\u0000other solutions based on state-of-the-art GPU libraries when simulating Spiking\u0000Neural P systems.","PeriodicalId":501347,"journal":{"name":"arXiv - CS - Neural and Evolutionary Computing","volume":"49 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2024-08-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141945575","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Marc Pickett, Aakash Kumar Nain, Joseph Modayil, Llion Jones
Modern machine learning systems have demonstrated substantial abilities with methods that either embrace or ignore human-provided knowledge, but combining benefits of both styles remains a challenge. One particular challenge involves designing learning systems that exhibit built-in responses to specific abstract stimulus patterns, yet are still plastic enough to be agnostic about the modality and exact form of their inputs. In this paper, we investigate what we call The Ungrounded Alignment Problem, which asks How can we build in predefined knowledge in a system where we don't know how a given stimulus will be grounded? This paper examines a simplified version of the general problem, where an unsupervised learner is presented with a sequence of images for the characters in a text corpus, and this learner is later evaluated on its ability to recognize specific (possibly rare) sequential patterns. Importantly, the learner is given no labels during learning or evaluation, but must map images from an unknown font or permutation to its correct class label. That is, at no point is our learner given labeled images, where an image vector is explicitly associated with a class label. Despite ample work in unsupervised and self-supervised loss functions, all current methods require a labeled fine-tuning phase to map the learned representations to correct classes. Finding this mapping in the absence of labels may seem a fool's errand, but our main result resolves this seeming paradox. We show that leveraging only letter bigram frequencies is sufficient for an unsupervised learner both to reliably associate images to class labels and to reliably identify trigger words in the sequence of inputs. More generally, this method suggests an approach for encoding specific desired innate behaviour in modality-agnostic models.
{"title":"The Ungrounded Alignment Problem","authors":"Marc Pickett, Aakash Kumar Nain, Joseph Modayil, Llion Jones","doi":"arxiv-2408.04242","DOIUrl":"https://doi.org/arxiv-2408.04242","url":null,"abstract":"Modern machine learning systems have demonstrated substantial abilities with\u0000methods that either embrace or ignore human-provided knowledge, but combining\u0000benefits of both styles remains a challenge. One particular challenge involves\u0000designing learning systems that exhibit built-in responses to specific abstract\u0000stimulus patterns, yet are still plastic enough to be agnostic about the\u0000modality and exact form of their inputs. In this paper, we investigate what we\u0000call The Ungrounded Alignment Problem, which asks How can we build in\u0000predefined knowledge in a system where we don't know how a given stimulus will\u0000be grounded? This paper examines a simplified version of the general problem,\u0000where an unsupervised learner is presented with a sequence of images for the\u0000characters in a text corpus, and this learner is later evaluated on its ability\u0000to recognize specific (possibly rare) sequential patterns. Importantly, the\u0000learner is given no labels during learning or evaluation, but must map images\u0000from an unknown font or permutation to its correct class label. That is, at no\u0000point is our learner given labeled images, where an image vector is explicitly\u0000associated with a class label. Despite ample work in unsupervised and\u0000self-supervised loss functions, all current methods require a labeled\u0000fine-tuning phase to map the learned representations to correct classes.\u0000Finding this mapping in the absence of labels may seem a fool's errand, but our\u0000main result resolves this seeming paradox. We show that leveraging only letter\u0000bigram frequencies is sufficient for an unsupervised learner both to reliably\u0000associate images to class labels and to reliably identify trigger words in the\u0000sequence of inputs. More generally, this method suggests an approach for\u0000encoding specific desired innate behaviour in modality-agnostic models.","PeriodicalId":501347,"journal":{"name":"arXiv - CS - Neural and Evolutionary Computing","volume":"111 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2024-08-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141945584","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：