GLDNN: A Group-Level Dynamic Neural Network via Knowledge Distillation for Image Recognition
摘要
In recent years, Dynamic Neural Networks (DNNs) have attracted significant attention as they can adapt to limited-resource platforms such as mobile devices and self-driving cars. DNNs can dynamically adjust their parameters and architectures through adaptive inference, reducing computational complexity and balancing accuracy and efficiency. This study presents the Group-Level Dynamic Neural Network (GLDNN), aiming to address the deployment challenges of DNNs in real-world scenarios. During training, this model constructs multiple subnets, each optimized for a different accuracy-efficiency trade-off. At deployment, the most suitable subnets can be selected according to the requirements of the target device, enabling efficient resource utilization and excellent performance. The experimental results on the FMNIST dataset are remarkable. Subnets A and B of the proposed model outperform the existing models by 50% and 70% respectively. Moreover, knowledge distillation training is employed to mitigate the accuracy decline caused by a high pruning rate, strongly demonstrating the effectiveness of the model.