SEM: A Novel Semantic Enhanced Multimodal Named Entity Recognition Method for Social Media
摘要
As social posts increasingly exhibit multimodal features, multimodal named entity recognition (MNER) tasks on social media play a crucial role in various downstream tasks, such as targeted advertising and social user targeting. However, existing MNER methods face challenges such as semantic gaps between text and images, and ambiguities in multimodal representations. To overcome these challenges, we propose a novel semantic enhanced multimodal entity recognition method, SEM. This method proposes to employ Transformer architecture and cross-modal attention mechanism to explore the semantic complementarity between textual and visual features. Specifically, we design a semantic enhancement method to strengthen external support for textual information. Additionally, we introduce a multimodal feature fusion encoding module that leverages three types of visual features to represent various aspects of image semantics. Finally, we integrate these features through collaborative representations and employ a shared multi-task label decoder to decode the textual and multimodal representations for entity prediction jointly. The superiority of the proposed model is demonstrated by the experimental findings on two public Twitter datasets.