On Stage-Wise Backpropagation for Improving Cheng’s Method for Fully Connected Cascade Networks

In this journal, Cheng has proposed a backpropagation (BP) procedure called BPFCC for deep fully connected cascaded (FCC) neural network learning in comparison with a neuron-by-neuron (NBN) algorithm of Wilamowski and Yu. Both BPFCC and NBN are designed to implement the Levenberg-Marquardt method, which requires an efficient evaluation of the Gauss-Newton (approximate Hessian) matrix \(\nabla \textbf{r}^\textsf{T} \nabla \textbf{r}\), the cross product of the Jacobian matrix \(\nabla \textbf{r}\) of the residual vector \(\textbf{r}\) in nonlinear least squares sense. Here, the dominant cost is to form \(\nabla \textbf{r}^\textsf{T} \nabla \textbf{r}\) by rank updates on each data pattern. Notably, NBN is better than BPFCC for the multiple \(q~\!(>\!1)\)-output FCC-learning when q rows (per pattern) of the Jacobian matrix \(\nabla \textbf{r}\) are evaluated; however, the dominant cost (for rank updates) is common to both BPFCC and NBN. The purpose of this paper is to present a new more efficient stage-wise BP procedure (for q-output FCC-learning) that reduces the dominant cost with no rows of \(\nabla \textbf{r}\) explicitly evaluated, just as standard BP evaluates the gradient vector \(\nabla \textbf{r}^\textsf{T} \textbf{r}\) with no explicit evaluation of any rows of the Jacobian matrix \(\nabla \textbf{r}\).