Exploring the Knowledge Distillation

Knowledge distillation is a well-known method of model compressing but it is largely under-used. Knowledge distillation has the ability to reduce the size of a model by transferring the knowledge from a large (pre-trained or not) model into a smaller one with minimum loss to the accuracy, for a better fit in edge devices and the devices with low computational power. Furthermore, it can be used also as a performance enhancer for the student model as we’ll see in the experiments since it doesn’t need any additional data to be effective. This paper argues that the knowledge distillation method or one of its variations should always be used on deep learning models. In our experiments we trained and tested our models on some well-known datasets (MNIST, CIFAR10 and CIFAR100) to prove that the knowledge distillation and two of its variations give positive results when applied on over-fitted or under-fitted teacher models.