AAKD-Net: Attention-Based Adversarial Knowledge Distillation Network for Image Classification

Deep neural networks have achieved remarkable success in various research fields, but they face limitations. Firstly, complex models are often required to handle challenging scenarios. Secondly, limited storage and processing power on mobile devices hinder model training and deployment. To address these challenges, we propose a novel approach using a compact and efficient student model to learn from a cumbersome teacher model. To enhance feature map information extraction, we introduce an attention structure that leverages the rich features in the teacher model’s feature maps. Adversarial training is incorporated by treating the student model as a generator and employing a discriminator to differentiate between teacher and student feature maps. Through an iterative process, the student model’s feature map gradually approximates that of the teacher while improving the discriminator’s discrimination abilities. By leveraging the knowledge of the teacher model and incorporating attention mechanisms and adversarial training, our approach provides a compelling solution to the challenges of complex model architectures and limited hardware resources. It achieves impressive performance enhancements with the student model.