Exploiting Symmetries of Distributions in CNNs and Folded Coding

Abstract

We introduce the concept of Folded Coding" for continuous univariate distributions estimating the distribution and coding the samples simultaneously. Folded Coding assumes symmetries in the distribution and requires significantly fewer parameters compared to conventional models when the symmetry assumption is satisfied. We incorporate the mechanics of Folded Coding into Convolutional Neural Networks (CNN) in the form of layers referred to as Binary Expanded ReLU (BEReLU) Shared Convolutions and Instance Fully Connected (I-FC). BEReLU and I-FC force the network to have symmetric functionality in the space of samples. Therefore similar patterns of prediction are applied to sections of the space where the model does not have observed samples. We experimented with BEReLU on generic networks using different parameter sizes on CIFAR-10 and CIFAR-100. Our experiments show increased accuracy of the models equipped with the BEReLU layer when there are fewer parameters. The performance of the models with BEReLU layer remains similar to original network with the increase of parameter number. The experiments provide further evidence that estimation of distribution symmetry is part of CNNs' functionality.