Data-driven weight initialization strategy for convolutional neural networks

Training deep Convolutional Neural Networks consists of multiple forward and backward passes over several epochs until the loss converges, making it time-consuming. Various factors affect the training times of networks, weight initialization being one of them. The idea of a proper weight initialization technique is to set the initial weights such that the network converges faster by extracting meaningful features from the data. This paper proposes a data-driven weight initialization to accelerate the training process. This technique is based on obtaining initial weights by appropriately analysing the training data. The proposed weight initialization strategy initializes filters from a pre-defined filter bank that is created before training, and it contains standard edge and texture feature extracting filters and data-driven filters obtained using principal component analysis, linear discriminant analysis and partial least squares. The proposed technique has been validated on AlexNet, VGG-16 and ResNet-50 for Intel Image Classification, CIFAR10 and CIFAR100 datasets. The results show that the proposed technique gives better validation results than other state-of-the-art techniques in fewer epochs and works well with state-of-the-art activation functions.