Quality Estimation (QE) is the prediction of translation quality without reference to the translation. Critical Error Detection (CED), as a sub-task of the QE task, aims to detect and identify significant meaning biases in machine translation that may cause serious damage. State-of-the-art CED models are supervised: they need to be trained on data obtained by multiple professionals labeled with specific CED labels based on the output of some machine translation system, thus making them dependent on the content of the training set, the scarcity of which and the non-uniformity in the distribution of the labels can affect performance. In order to solve the above problems, in this paper, we propose a training-free CED approach-CED based on anchors test, our approach only needs to test the anchors in the source language by cross-lingual masking test, and then determine whether the translation contains critical errors based on the test results. We conducted experiments on blind test data from the WMT2021 CED sharing task using different models. The results show the effectiveness of our proposed method, reaching the performance of the supervised baseline on En-Zh, but overall there is still a gap compared to supervised CED, and we give an example to illustrate the potential of our method for explainable applications.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Critical Error Detection Based on Anchors Test

  • Kaiyuan Huang,
  • Junguo Zhu

摘要

Quality Estimation (QE) is the prediction of translation quality without reference to the translation. Critical Error Detection (CED), as a sub-task of the QE task, aims to detect and identify significant meaning biases in machine translation that may cause serious damage. State-of-the-art CED models are supervised: they need to be trained on data obtained by multiple professionals labeled with specific CED labels based on the output of some machine translation system, thus making them dependent on the content of the training set, the scarcity of which and the non-uniformity in the distribution of the labels can affect performance. In order to solve the above problems, in this paper, we propose a training-free CED approach-CED based on anchors test, our approach only needs to test the anchors in the source language by cross-lingual masking test, and then determine whether the translation contains critical errors based on the test results. We conducted experiments on blind test data from the WMT2021 CED sharing task using different models. The results show the effectiveness of our proposed method, reaching the performance of the supervised baseline on En-Zh, but overall there is still a gap compared to supervised CED, and we give an example to illustrate the potential of our method for explainable applications.