Improving Sign Language Gloss Translation with Low-Resource Machine Translation Techniques

A cascaded Sign Language Translation system first maps sign videos to gloss annotations and then translates glosses into spoken language text. This chapter focuses on the second stage, gloss translation, which is challenging due to the scarcity of publicly available parallel data. We approach gloss translation as a low-resource machine translation task and investigate several popular methods for improving translation quality, including hyperparameter search, pretrained multilingual models, rule-based data augmentation, back translation, and curriculum learning. We discuss the potentials and pitfalls of these methods based on experiments conducted on a German Sign Language dataset, RWTH-PHOENIX-Weather 2014T, and a Chinese Sign Language dataset, CSL Daily. We also show how a sign-to-text translation system would benefit from the initialization of gloss-to-text checkpoints obtained with different methods. We further conduct word-level error analyses to study how the two components of the sign translation system contribute to the translation errors, respectively.