On Stage-Wise Backpropagation for Improving Cheng’s Method for Fully Connected Cascade Networks

In this journal, Cheng has proposed a backpropagation (BP) procedure called BPFCC for deep fully connected cascaded (FCC) neural network learning in comparison with a neuron-by-neuron (NBN) algorithm of Wilamowski and Yu. Both BPFCC and NBN are designed to implement the Levenberg-Marquardt method, which requires an efficient evaluation of the Gauss-Newton (approximate Hessian) matrix  \(\nabla \textbf{r}^\textsf{T} \nabla \textbf{r}\) $\mathrm{\nabla }{\mathbf{r}}^{\mathsf{T}}\mathrm{\nabla }\mathbf{r}$ , the cross product of the Jacobian matrix  \(\nabla \textbf{r}\) $\mathrm{\nabla }\mathbf{r}$ of the residual vector  \(\textbf{r}\) $\mathbf{r}$ in nonlinear least squares sense. Here, the dominant cost is to form  \(\nabla \textbf{r}^\textsf{T} \nabla \textbf{r}\) $\mathrm{\nabla }{\mathbf{r}}^{\mathsf{T}}\mathrm{\nabla }\mathbf{r}$ by rank updates on each data pattern. Notably, NBN is better than BPFCC for the multiple \(q~\!(>\!1)\) $q(>1)$ -output FCC-learning when q rows (per pattern) of the Jacobian matrix  \(\nabla \textbf{r}\) $\mathrm{\nabla }\mathbf{r}$ are evaluated; however, the dominant cost (for rank updates) is common to both BPFCC and NBN. The purpose of this paper is to present a new more efficient stage-wise BP procedure (for q-output FCC-learning) that reduces the dominant cost with no rows of \(\nabla \textbf{r}\) $\mathrm{\nabla }\mathbf{r}$ explicitly evaluated, just as standard BP evaluates the gradient vector \(\nabla \textbf{r}^\textsf{T} \textbf{r}\) $\mathrm{\nabla }{\mathbf{r}}^{\mathsf{T}}\mathbf{r}$ with no explicit evaluation of any rows of the Jacobian matrix  \(\nabla \textbf{r}\) $\mathrm{\nabla }\mathbf{r}$ .