MSMD: A Multi-Stage Meta Distillation Strategy for Cross-Lingual Natural Language Understanding

Multilingual pre-trained models provide a universal language representation for cross-lingual natural language understanding tasks, reducing effort required for training individual languages and achieving good generalization. However, replacing encoder part with multilingual pre-trained models to extend their applicability to cross-lingual domains does not yield optimistic results on multiple low-resource language corpora. To address this, we propose Multi-Stage Meta Distillation (MSMD), a strategy that integrates optimization-based meta learning and technique of knowledge transfer through soft-label distillation. This approach leverages advantages of meta learning in few-shot training methods and employs knowledge distillation to facilitate transfer of soft-label knowledge, compressing multilingual models, improving inference speed, and mitigating the loss of performance. Experiments on various benchmarks demonstrate that MSMD achieves significant improvements compared to multiple natural language understanding models. Furthermore, it exhibits lower sensitivity to different choices of student model capacity and hyperparameters, facilitating application of this algorithm for model compression across different tasks and models.