Scaling Bit-Flexible Neural Networks

Abstract

This paper proposes a neural network training scheme in order to obtain the network weights represented in the fixed-point number format such that under the different truncated lengths of the weights, our neural new network can all achieve near-optimized inference accuracy at the corresponding word-length. The similar idea has been explored before; however, the salient feature of our proposed scaling bit-progressive method is we have further taken into account the use and training of weight scaling factor, which can significant improve the inference accuracy. Our experimental results show that our trained Resnet- 18 neural network can improve the top-1 and top-5 accuracies of Tiny-ImageNet dataset by the average of 11.02% and 9.21% compared with the previous work without using the scaling factor. The top-1 and top-5 accuracy losses compared with float-point weights are only about 0.5% and 0.31% under the truncated size of 5-bit. The proposed method can be applied for neural network accelerators especially for those which support bit-serial processing.