A conceptual framework for implementing neural networks on massively parallel machines

Abstract

This paper describes a framework for implementing neural networks on massively parallel machines. The framework is generic and applies to a range of neural networks (Multi Layer Perceptron, Competitive Learning, Self-Organising Map, etc.) as well as a range of massively parallel machines (Connection Machine, Distributed Array Processor, MasPar). It consists of two phases: an abstract decomposition of neural networks and a machine specific decomposition. The abstract decomposition identifies the parallelism implemented by neural networks, and provides alternative distribution schemes according to the required exploitation of parallelism. The machine specific decomposition considers the relevant machine criteria, and integrates these with the result of the abstract decomposition to form a 'decision' system. This system formalises the relative gain of each distribution scheme according to neural network and machine criteria. It then identifies their possible optimisations. Finally, it computes and ranks the absolute speed up of each distribution scheme.<>