Sine and cosine based learning rate for gradient descent method

Abstract

Deep learning networks have been trained using first-order-based methods. These methods often converge more quickly when combined with an adaptive step size, but they tend to settle at suboptimal points, especially when learning occurs in a large output space. When first-order-based methods are used with a constant step size, they oscillate near the zero-gradient region, which leads to slow convergence. However, these issues are exacerbated under nonconvexity, which can significantly diminish the performance of first-order methods. In this work, we propose a novel Boltzmann Probability Weighted Sine with a Cosine distance-based Adaptive Gradient (BSCAGrad) method. The step size in this method is carefully designed to mitigate the issue of slow convergence. Furthermore, it facilitates escape from suboptimal points, enabling the optimization process to progress more efficiently toward local minima. This is achieved by combining a Boltzmann probability-weighted sine function and cosine distance to calculate the step size. The Boltzmann probability-weighted sine function acts when the gradient vanishes and the cooling parameter remains moderate, a condition typically observed near suboptimal points. Moreover, using the sine function on the exponential moving average of the weight parameters leverages geometric information from the data. The cosine distance prevents zero in the step size. Together, these components accelerate convergence, improve stability, and guide the algorithm toward a better optimal solution. A theoretical analysis of the convergence rate under both convexity and nonconvexity is provided to substantiate the findings. The experimental results from language modeling, object detection, machine translation, and image classification tasks on a real-world benchmark dataset, including CIFAR10, CIFAR100, PennTreeBank, PASCALVOC and WMT2014, demonstrate that the proposed step size outperforms traditional baseline methods.