Bayesian Optimization with Time-Decaying Jitter for Hyperparameter Tuning of Neural Networks

This paper introduces a modification of the ordinary Bayesian optimization algorithm for hyperparameter tuning of neural networks. The proposed algorithm utilizes the time-decaying parameter ξ (jitter) to dynamically balance between exploration and exploitation. This algorithm is compared with the ordinary Bayesian optimization algorithm with various constant values of ξ; for that, diverse artificial landscapes were used. In this comparison, the proposed algorithm shows a better performance for some artificial landscapes and numbers of dimensions of the search domain. For some others, the ordinary algorithm outperforms the proposed one, but in most cases, there is no statistically significant difference between the two algorithms. Both algorithms then are used to tune hyperparameters of a neural network for semantic image segmentation. The corresponding analysis has shown that both algorithms give a comparable performance.