Multimodal Named Entity Recognition with Synthesized SVG Graphics and Structural Semantic Consistency Scoring
摘要
Named entity recognition (NER) is a fundamental task in natural language processing (NLP). However, it often has difficulty in handling entity ambiguity due to limited contextual clues. To address this challenge, traditional multimodal methods introduce social media images to assist text understanding, but the semantic deviation and noise of images seriously restrict the multimodal modeling effect. This paper proposes NER-S, a novel multimodal NER framework that integrates text-guided Scalable Vector Graphics (SVG) as reliable visual information and combines structural semantic consistency score (SSCS) to select images with high visual and semantic consistency as auxiliary information to improve entity recognition performance. Specifically, the original text is first input into the SVG image generation model to generate candidate images. Then, the optimal image is selected by SSCS and input into the multimodal named entity recognition model as the final visual supplement. Experiments on Twitter-2015 and Twitter-2017 datasets demonstrate the effectiveness of NER-S, with F1 scores of 76.60% and 86.87%, respectively. Our model outperforms all text-only baselines and exhibits comparable or superior robustness and generalization capabilities to existing multimodal models with real-world images.