Dealing with Training Deficiencies

Basic techniques to define and use training data for network optimization may not produce the desired optimum because of the complexity of the loss function, deficiencies of the gradient descent method, and the scarcity of training data. Advanced methods and strategies deal with these problems. First, advanced data augmentation techniques are presented to increase the robustness of the classifier. Occlusion augmentation trains the network to consider a wider range of features for classification. Adding noise makes the decision less dependent on small variation of the input data. Adversarial training does the same but for variation of features toward partitions in feature space. Generative models produce new training data from a learned distribution. Unlabeled samples may be added to training data for semi-supervised training. Then, various methods to improve training are discussed. Transfer learning exploits existing classification problems to solve a new problem. Weight regularization as well as batch and weight normalization explore effects of normalizing weights deep in the network to arrive at a smoother loss surface. Ensemble learning uses a group of simple classifiers instead of a single complex classifier. Residual links are presented to solve the problem of a vanishing gradient in very deep networks.