Hyper-parameter Tuning

Hyper-parameters can be loosely defined as those parameters that are not changed during the training process. For example, number of layers in a FFNN, number of neurons in each layer, activation functions, learning rate, and so on. In this chapter I will discuss the problem of finding the best hyper-parameters to get the best results from your models. Doing this is called hyper-parameter tuning. I will first describe what a black-box optimisation problem is and how those classes of problems relate to hyper-parameter tuning. I will discuss the two most known methods to tackle these kind of problems: grid search and random search. You will understand, with examples, which one works under which conditions, and a few tricks that are very helpful, such as coarse to fine optimisation and sampling on a logarithmic scale. At the end of the chapter, you should know what hyper-parameter tuning is, why it is important, and how it works.