A Task-Efficient Gradient Guide Knowledge Distillation for Pre-train Language Model Compression

In knowledge distillation, our aim is to effectively transfer knowledge from large models (teachers) to smaller models (students). However, due to the limited learning capacity of student models, previous studies often overlook the importance of gradient information from teacher models, which can significantly enhance model performance. Despite its effectiveness, the excessive redundancy of this gradient information can also lead to overfitting. In this study, we propose a novel gradient-guided strategy that integrates the teacher model’s gradients into the distillation process of student models, helping them to more accurately learn the key information and features that the teacher models focus on. We designed a task-oriented multilayer perceptron (MLP) gradient extraction strategy and incorporated this gradient information into the distillation loss to guide the student models. Experiments show that this method achieves a score of 87.91% on the English dataset of the GLUE benchmark, an improvement of 0.91% over the previous gradient method. Further analysis reveals that this strategy makes the student model’s gradients more consistent with those of the teacher model, significantly enhancing the model’s interpretability.