Scholarlink

<p>Knowledge distillation is an effective technique for compressing neural machine translation models. However, traditional word-level knowledge distillation only transfers word-level probabilities. This leads to a problem: the student model’s encoder cannot effectively model the syntactic structure of the source language. Consequently, this may cause structural disarray in the translated output. Integrating prior syntactic information and selectively learning valuable knowledge can significantly enhance neural machine translation performance.To address this, we propose a selective self-attention knowledge distillation method with encoder-side dependency constraints. The method transforms a source-language dependency syntax tree into an adjacency matrix. This matrix is then applied to the weighted computation of the self-attention network. Furthermore, the method introduces a selective distillation strategy. This strategy identifies and transfers knowledge from the teacher model that possesses greater training utility. Additionally, explicit syntactic constraints are embedded into the self-attention mechanisms of both the teacher and student models. Structural alignment distillation is employed to acquire cross-lingual alignment knowledge. This integrated approach enhances the distillation effect.Experimental results on multiple translation tasks demonstrate the effectiveness of our method. The tasks include WMT’14 En-De, IWSLT’14 En-De/En-Fr/En-Ar, and IWSLT’15 En-Vi. Our method effectively improves translation quality. For instance, on WMT’14 En-De, our approach achieves improvements of +1.57 BLEU and +8.19 COMET over the Transformer-Base model, with particularly significant gains on long sentences. The joint application of dependency-constrained self-attention and selective knowledge distillation effectively enhances machine translation performance.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Dependency-constrained selective knowledge distillation with self-attention for neural machine translation

  • 期刊论文

  • 发表日期：

    2026年7月1日

查看全文
  • EnLiang Liu,
  • ZhenHan Wang,
  • ShengXiang Gao,
  • Zhengtao Yu,
  • JunJun Guo

摘要

Knowledge distillation is an effective technique for compressing neural machine translation models. However, traditional word-level knowledge distillation only transfers word-level probabilities. This leads to a problem: the student model’s encoder cannot effectively model the syntactic structure of the source language. Consequently, this may cause structural disarray in the translated output. Integrating prior syntactic information and selectively learning valuable knowledge can significantly enhance neural machine translation performance.To address this, we propose a selective self-attention knowledge distillation method with encoder-side dependency constraints. The method transforms a source-language dependency syntax tree into an adjacency matrix. This matrix is then applied to the weighted computation of the self-attention network. Furthermore, the method introduces a selective distillation strategy. This strategy identifies and transfers knowledge from the teacher model that possesses greater training utility. Additionally, explicit syntactic constraints are embedded into the self-attention mechanisms of both the teacher and student models. Structural alignment distillation is employed to acquire cross-lingual alignment knowledge. This integrated approach enhances the distillation effect.Experimental results on multiple translation tasks demonstrate the effectiveness of our method. The tasks include WMT’14 En-De, IWSLT’14 En-De/En-Fr/En-Ar, and IWSLT’15 En-Vi. Our method effectively improves translation quality. For instance, on WMT’14 En-De, our approach achieves improvements of +1.57 BLEU and +8.19 COMET over the Transformer-Base model, with particularly significant gains on long sentences. The joint application of dependency-constrained self-attention and selective knowledge distillation effectively enhances machine translation performance.