Multi-Task Feature Self-Distillation for Semi-Supervised Machine Translation

The performance of the model suffers dramatically when the scale of supervision data is constrained because the most sophisticated neural machine translation models are all data-driven techniques. Adding monolingual data to the model training process and using the back-translation method to create pseudo-parallel sentences for data augmentation is the current accepted approach to address data shortage. However, this method’s training procedures are laborious, it takes a lot of time, and some of the created pseudo-parallel sentences are of poor quality and detrimental to the model. In this paper, we propose a semi-supervised training method—Multi-Task Feature Self-Distillation (MFSD), which can train models online on a mixed dataset consisting of bilingual and monolingual data jointly. Based on the supervised machine translation task, we propose a self-distillation task for the encoder and decoder of the source and target languages to train two kinds of monolingual data online. In the self-distillation task, we build a teacher model by integrating the student models of the previous rounds, and then use the feature soft labels output by the teacher model with more stable performance to guide the student model training online, and realize the online mining single High-level knowledge of language data by comparing the consistency of the output features of the two models. We conduct experiments on the standard data set and the multi-domain translation data set respectively. The results show that MFSD performed better than mainstream semi-supervised and data augmentation methods approaches. Compared with supervised training, our method improves the BLEU score by an average of +2.27, and effectively improves the model’s domain adaptability and generalization ability.