Reliability of Effort Indicators for Human Translation and Post-editing of Machine Translation: An Eye Tracking and Keylogging Study Involving Student Translators

Evaluating student translators’ effort in translation and post-editing of machine translation provides valuable reference for translation training and learning. The effort indicators currently used such as those based on eye-tracking and key-logging data are variant. This study investigates the reliability of those effort indicators. It focuses on student translators working on the English-Chinese language pair. An eye-tracking and key-logging experiment was conducted to capture participants’ fixation, pause, translation time, and edit data. Thirty-three students participated in the experiment and did translation and post-editing tasks. Data analyses have revealed the following characteristics of effort indicators: indicators based on the source text (ST) word count are more reliable than those based on the target text (TT) word count; indicators based on pause count are more reliable than those based on pause duration; indicators based on ST fixation are more reliable than those based on TT fixation; and the indicator of edit number is applicable to post-editing but not human translation. The findings are intended to facilitate the more precise evaluation of student translators’ effort.