Imaginations Generate Images for Multi-modal Machine Translation

Multi-modal machine translation (MMT) aims at exploring better translation systems by integrating the visual annotation which presents the content described in the bilingual parallel sentence pair into the conventional only-text neural machine translation (NMT). However, existing methods heavily rely on the manual annotated images data set. The cost of manual image annotation is relatively high at this stage. In this paper, we propose the generative imagination network with transformer to automatically generate visual annotations semantic-equivalent with source and target sentences. The proposed model receives the inputs of source-target bilingual sentences and generates visual annotations for MMT. Experiments analysis demonstrate that our model can generate high-quality annotated images and prompt the performance of MMT. Additionally, we use our model to generate annotated images for a famous English-German IWSLT-2015, the experimental results show the improvement for MMT.