Fractional-order gradient approach for optimizing neural networks: A theoretical and empirical analysis

Abstract

This article proposes a modified fractional gradient descent algorithm to enhance the learning capabilities of neural networks, comprising the benefits of a metaheuristic optimizer. The use of fractional derivatives, which possess memory properties, offers an additional degree of adaptability to the network. The convergence of the fractional gradient descent algorithm, incorporating the Caputo derivative in the neural network’s backpropagation process, is thoroughly examined, and a detailed convergence analysis is provided which indicates that it enables a more gradual and controlled adaptation of the network to the data. Additionally, the optimal fractional order has been found for each dataset, a contribution that has not been previously explored in the literature, which has a significant impact on the training of neural networks with fractional gradient backpropagation. In the experiments, four classification datasets and one regression dataset were used, and the results consistently show that the proposed hybrid algorithm achieves faster convergence across all cases. The empirical results with the proposed algorithm are supported by theoretical convergence analysis. Empirical results demonstrate that the proposed optimizer with optimal order yields more accurate results compared to existing optimizers.