DRA: dynamic routing attention for neural machine translation with low-resource languages

In recent years, the utilization of deep models has significantly enhanced the performance of neural machine translation (NMT). Nevertheless, the uneven distribution of data leads to critical challenges. Specifically, low-frequency words severely affect translation performance. This is especially in low-resource language translation, where the training of low-frequency words is inadequate. To address this issue, we use syntactic and word frequency information to enhance the performance of encoding representations of input sequence. we propose a simple approach called Dynamic Routing Attention (DRA). When processing different words, DRA dynamically adjusts the Self-attention weight based on word frequency and source syntactic, which enables the encoder Self-attention to focus on the surrounding words and the words with syntactic associations rather than the current word solely. Consequently, our method improves the representation capability of the encoder in processing sentences containing low-frequency words. Using Transformer RPR model as a baseline model, we demonstrate the effectiveness of our method with the experiments on machine translation tasks of WMT14 English-German, IWLST14 English-German, IWLST14 English-Vietnamese, and TED talk Thai-Chinese. Using our DRA method achieve a improvement of 0.79–2.81 BLEU score on the four tasks.