A Simulated Annealing BP Algorithm for Adaptive Temperature Setting

The backpropagation (BP) algorithm is so far, the most effective algorithm for neural network learning and training. This algorithm is based on the fast gradient descent method in optimization theory, which is essentially a deterministic local optimization algorithm. In this paper, for improving its local optimization, enhancing its global optimization potential ability is discussed by combining with the simulated annealing mechanism. The BP algorithm is utilized as the core component under the framework of simulated annealing algorithm, and the Metropolis criterion is used as the heuristic strategy. Integrating advantages of the BP algorithm and the simulated annealing algorithm, An adaptive simulated annealing based BP algorithm is designed. In this algorithm there two adaptive strategies, one is the setting of annealing temperature, and the other is the random perturbation of connection weight and bias value. The adaptive temperature setting can simultaneously reflect changes of the loss function value, and the adaptive perturbation is related to the network complexity and loss function value. By using the small batch gradient descent method used in machine learning commonly, the update of the annealing temperature is synchronized with the input sample batch, which follows up the search state of the algorithm running. The perturbed renewal of those parameters of the neural network is based on the current network configuration to avoid blind random perturbation. The algorithm is tested with the Fashion-MNIST and CIFAR-10 datasets, which are widely used in the current deep learning research field. The classification and recognition accuracies of the algorithm exceed 99% both. These results are better than the best results reported currently for the two datasets. In addition, the adaptive BP algorithm is repeated 30 times with these two datasets respectively. The statistical analysis results show the robustness and stability of the algorithm.