MIMD implementation of neural networks through pipelined, parallel communication trees

Abstract

Hardware implementations and single-instruction-stream/multiple-data-stream (SIMD) simulations are relatively inflexible, while multiple-instruction-stream (MIMD) simulations often trade their flexibility for increased efficiency. An alternative technique is presented which is based on pipelining fewer but larger messages through parallel, broadcast/accumulate trees. This method exploits both the structural parallelism of neural networks and the data parallelism of neural algorithms. The mapping is flexible to changes in the network architecture and learning algorithm and is suited to a variety of computer configurations. Experimental results show a higher efficiency than similar implementation.<>