The link between translation difficulty and the quality of machine translation: a literature review and empirical investigation

We survey the relevant literature on translation difficulty and automatic evaluation of machine translation (MT) quality and investigate whether source text’s translation difficulty features contain any information about MT quality. We analyse the 2017–2019 Conferences on Machine Translation (WMT) data of machine translation quality of English news text translated to eleven different languages (Chinese, Czech, Estonian, Finnish, Latvian, Lithuanian, German, Gujarati, Kazakh, Russian, and Turkish). We find (weak) negative correlation between the source text’s length, polysemy and structural complexity and the corresponding human evaluated quality of machine translation. This suggests a potentially important but measureable influence of source text’s translation difficulty on MT quality.