Basic Network Training for Image Classification

We review basic aspects of network training and explore the influence of different decisions on the training success. An important hyperparameter for network training is the number of epochs during which network weights are optimized. We will discuss how early stopping weight optimization is used to avoid overfitting to the training data. Parametrization of further hyperparameters besides early stopping that we already encountered will be discussed as well. It follows a detailed discussion about basic decisions for successful network training. We cover the selection of initial weights, the choice of a suitable loss function, and the choice and parameterization of the gradient descent method. Label smoothing of the ground truth data will be discussed as well since the way in which output is generated never produces the 1- and 0-values of a one-hot-vector. The chapter is concluded with a discussion about the interpretation of training and validation loss curves. Ideal curves would monotonically decrease until reaching a minimum. Various problems caused by network training will result in deviations from this ideal in different ways.