Improving Image Classification Efficiency with Knowledge Distillation and Channel Attention
摘要
Image classification is a fundamental task in computer vision, with convolutional neural networks (CNNs) being the state-of-the-art approach for this task. However, CNNs can be large and computationally expensive, which limits their deployment on resource-constrained devices. To address this issue, model optimization and compression techniques such as knowledge distillation and channel attention have been proposed. In this paper, we propose a modified version of the skip-connection knowledge distillation method, which integrates the efficient channel attention module. Our experiments on the CIFAR-10 dataset show that this approach improves the performance of both large teacher-small student and self-distillation scenarios, and also leads to smaller model sizes. We also visualize the intermediate feature maps to understand the learning process of the network. Our results demonstrate the effectiveness of combining knowledge distillation and channel attention for improving the performance and efficiency of CNNs.