ATCBBC: A Novel Optimizer for Neural Network Architectures

For deep neural networks, gradient descent (GD) is the backbone. Slow convergence is an issue with GD. Using momentum is the well-known method of overcoming delayed convergence. GD’s learning factor is considerably improved by momentum. Several strategies, including Adam, diffGrad, and AngularGrad, have notably been put out to better manage the momentum for optimization toward the global minima (GM). Adam reduces momentum by reducing it through the square root of the moving average of the square of the past gradients, or the second moment. Sometimes the gradient overshoots the optimal solution and settles at the local minima (LM) as a result of the second moment’s abrupt decline. The friction factor in DiffGrad, which reduces this issue, is dependent on the difference between Adam’s current gradient and its recent gradient. Slow convergence is caused by the frictional constant, which further reduces the momentum. AngularGrad adapts the step size using smallest tangent values of the angles of two consecutive gradient. The novel optimization method described in this research, called ATCBBC, or adaptive term calculation, only uses bias correction, does away with the overshooting gradient issue in Adam, provides a smooth trajectory, and outperforms conventional optimizers in terms of classification accuracy on real-world datasets.