Optimizing Deep Learning Methods in Neural Network Architectures

Deep neural networks are a powerful tool for computer-assisted learning and have achieved significant success in numerous computer vision and image processing tasks. This paper discusses several new neural network structures that have better performance than the traditional feedforward neural network structure. A method of network structure optimization based on gradient descent and heavy-ball algorithms has been proposed. Furthermore, an approach based on the concept of sparse representation for simultaneous training and optimizing the network structure has been presented. According to CIFAR-10 and CIFAR-100 dataset classification tasks and experimental results, the optimization of ResNet and DenseNet structures using gradient descent and heavy-ball algorithms, accordingly, has been shown to result in better performance with increased depth of neural network. A neural network based on a sparse representation is shown to have the highest performance in all datasets. This strategy encourages quick data adaptation at each iteration. The results obtained can be used to design deeper neural networks with no loss of precision and computing speed.