An Alignment and Matching Network with Hierarchical Visual Features for Multimodal Named Entity and Relation Extraction

In this paper, we study the tasks of multimodal named entity recognition (MNER) and multimodal relation extraction (MRE), both of which involve incorporating visual modality to complement text modality. The core issues are how to bridge the modality gap and reduce modality noise. To address the first issue, we introduce an image-text alignment (ITA) module to obtain a better unimodal representation by aligning the inconsistent representations between image and text, which come from different encoders. To tackle the second issue, we propose an image-text matching (ITM) module that constructs hard negatives to improve the model's ability to capture the semantic correspondence between text and image. Besides, we also selectively combine and concatenate the hierarchical visual features obtained from both global and visual objects of Vision Transformer (ViT) as improved visual prefix for modality fusion. We conduct extensive experiments to demonstrate the effectiveness of our method (AMNet) (Code is available in https://github.com/Grident/AMNet ) and achieve state-of-the-art performance on three benchmark datasets.