Biomedical Named Entity Recognition Model Based on Knowledge Distillation

As the literature in the biomedical field continues to grow, the identification of biomedical named entities becomes increasingly important. A pre-trained language model called BioBERT was created exclusively for identifying biological named entities, and the performance of using the BioBERT model in extracting biomedical named entity identification has been dramatically improved compared to the previously proposed BERT model. However, due to its large model and many parameters, even more than 110 million. Its drawback is that it is time-consuming and requires significant resources. Consequently, we suggest a knowledge distillation strategy in this study, in which knowledge from a teacher model with a complicated structure is learned by a student model with a simple structure to increase recognition performance. In this paper, the BioBERT model is used as the teacher model and the BiLSTM model is used as the student model, and the best distillation effect is finally found when the weighting factor a = 0.3 by experimentally comparing different weighting factors. At this time, the F1 value of the refined student model is improved by 0.29% compared with the original model.