Exploring Interpretable Semantic Alignment for Multimodal Machine Translation

Multimodal Machine Translation (MMT) aims to introduce contextual information from other modalities, mainly vision, in order to enhance translation results. Existing methods focus on constructing the global cross-modal interaction between text and vision while ignoring the local semantic correspondences, which may improve the interpretability of multimodal feature fusion. To this end, we propose a novel multimodal fusion encoder with local semantic alignment for MMT. First, the multimodal alignment is modeled as a learnable Gaussian distribution that represents the correlation of image and text between spatial feature and the central position. Then, a dynamic fusion gate is developed to refine the multimodal context representations. Finally, these vectors are utilized for sentence translation by a cross attention of the decoder. We conduct experiments on the Multi30K dataset. In-depth analysis of the results demonstrates the effectiveness and interpretability of our model, which is highly competitive compared to the baseline. Further exploration of extractors in MMT shows that a large multimodal pre-trained model can provide more fine-grained semantic alignment, thus giving it an advantage over general integrative MMT methods.