Fusion based Heterogeneous Convolutional Neural Networks Architecture

Abstract

In recent years, Deep Convolutional Neural Networks (DCNNs) have gained lots of attention and won many competitions in machine learning, object detection, image classification, and pattern recognition. The breakthroughs in the development of graphical processing units have made it possible to train DCNNs quickly for state-of-the-art tasks such as image classification, speech recognition, and many others. However, to solve complex problems, these multilayered convolutional neural networks become increasingly large, complex, and abstract. We propose methods to improve the performance of neural networks while reducing their dimensionality, enabling a better understanding of the learning process. To leverage the extensive training, as well as strengths of several pretrained models, we explored new approaches for combining features from fully connected layers of models with heterogeneous architectures. The proposed approach combines features extracted from the penultimate fully connected layer from three different DCNNs. We merge the features of all three DCNNs together and apply principal component analysis or linear discriminant analysis. Our approach aims to reduce the dimensionality of the feature vector and find the smallest feature vector dimension that can maintain the classifier performance. For this task we use a linear Support Vector Machine as a classifier. We also investigate whether it is advantageous to fuse only penultimate fully connected layers, or to perform fusion based on other fully connected layers using multiple homogenous or heterogeneous networks. The results show that the fusion method outperformed both individual networks in terms of accuracy and computational time in all of our various trial sizes. Overall our fusion methods are faster and more accurate than individual networks in both training and testing. Finally, we compared heterogeneous with homogenous fusion methods and the results show heterogeneous methods outperform homogeneous methods.