A purely spiking approach to reinforcement learning

Abstract

At present, implementation of learning mechanisms in spiking neural networks (SNN) cannot be considered as a solved scientific problem despite plenty of SNN learning algorithms proposed. It is also true for SNN implementation of reinforcement learning (RL), while RL is especially important for SNNs because of its close relationship to the domains most promising from the viewpoint of SNN application such as robotics. In the present paper, an SNN structure is described which, seemingly, can be used in wide range of RL tasks. The distinctive feature of our approach is usage of only the spike forms of all signals involved — sensory input streams, output signals sent to actuators and reward/punishment signals. Besides that, selection of the neuron/plasticity models was determined by the requirement that they should be easily implemented on modern neurochips. The SNN structure considered in the paper includes spiking neurons described by a generalization of the LIFAT (leaky integrate-and-fire neuron with adaptive threshold) model and a simple spike timing dependent synaptic plasticity model (a generalization of dopamine-modulated plasticity). In this study, we use the model-free approach to RL but it is based on very general assumptions about RL task characteristics and has no visible limitations on its applicability (inside the class of model-free RL tasks). To test our SNN, we apply it to a simple but non-trivial task of training the network to keep a chaotically moving light spot in the view field of an emulated Dynamic Vision Sensor (DVS) camera. Successful solution of this RL problem can be considered as an evidence in favor of efficiency of our approach.