Training Low-Latency Deep Spiking Neural Networks with Knowledge Distillation and Batch Normalization Through Time

Abstract

Spiking Neural Networks (SNNs) can significantly enhance energy efficiency on neuromorphic hardware due their sparse, biological plausibility and binary event (or spike) driven processing. However, from the non-differentiable nature of a spiking neuron, training high-accuracy and low-latency SNNs is challenging. Recent researches continue to look for ways to improve accuracy and latency. To address these issues in SNNs, we propose a technique that concatenates Knowledge Distillation (KD) and Batch Normalization Through Time (BNTT) method in this study. The BNTT boosts low-latency and low-energy training in SNNs by allowing a neuron to handle the spike rate through various timesteps. The KD approach effectively transfers hidden information from the teacher model to the student network, which converts artificial neural network parameters to SNN weights. This concept allows enriching the performance of SNNs better than the prior technique. Experiments are carried out on the Tiny-ImageNet, CIFAR-10, and CIFAR-100 datasets. on various VGG architectures. We reach top-1 accuracy of 55.67% for ImageNet on VGG-11 and 73.11% for the CIFAR-100 dataset on VGG-16. These results demonstrate that our proposal outperforms earlier converted SNNs in accuracy with only 5 timesteps.