Multimodal Machine Translation Based on Text-Image Aggregation
摘要
Multimodal Neural Machine Translation (MNMT) aims to enhance Neural Machine Translation (NMT) models by integrating additional modalities. Although the performance of NMT can be enhanced by incorporating visual information, multimodal neural translation models still tend to prioritize textual information. Furthermore, their effectiveness heavily depends on the availability of a bilingual parallel corpus with manually labeled images. In this paper, we propose a generalized visual representation to assess the relevance of text to images, thereby overcoming the limitations of text annotation methods for manually labeled images. This approach allows plain text NMT models to directly leverage associated visual information without requiring annotation. Features are extracted using various methods, and cosine similarity is employed to identify the image that best matches the text. The CVAE is adopted to learn the information embedded in visual features associated with sentences. The method was evaluated on four widely used datasets: Multi30K English-to-German, Multi30K English-to-French, WMT22 English-to-Ukrainian, and WMT22 Ukrainian-to-English. The evaluation results demonstrate that the method achieves significant improvements over strong baselines. In the English-German translation task, the model achieved improvements of 1.64 BLEU on Test 2016 and 1.6 BLEU on Test 2017. In the English-French task, improvements were 1.07 BLEU on Test 2016 and 0.69 BLEU on Test 2017. The proposed method significantly enhances the construction efficiency of multimodal machine translation datasets, thereby positively influencing the advancement of research in the field of multimodal machine translation.