Named entity recognition (NER) is a fundamental task in natural language processing (NLP). However, it often has difficulty in handling entity ambiguity due to limited contextual clues. To address this challenge, traditional multimodal methods introduce social media images to assist text understanding, but the semantic deviation and noise of images seriously restrict the multimodal modeling effect. This paper proposes NER-S, a novel multimodal NER framework that integrates text-guided Scalable Vector Graphics (SVG) as reliable visual information and combines structural semantic consistency score (SSCS) to select images with high visual and semantic consistency as auxiliary information to improve entity recognition performance. Specifically, the original text is first input into the SVG image generation model to generate candidate images. Then, the optimal image is selected by SSCS and input into the multimodal named entity recognition model as the final visual supplement. Experiments on Twitter-2015 and Twitter-2017 datasets demonstrate the effectiveness of NER-S, with F1 scores of 76.60% and 86.87%, respectively. Our model outperforms all text-only baselines and exhibits comparable or superior robustness and generalization capabilities to existing multimodal models with real-world images.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Multimodal Named Entity Recognition with Synthesized SVG Graphics and Structural Semantic Consistency Scoring

  • Shujun Xia,
  • Yulong Zhou,
  • Wei Li

摘要

Named entity recognition (NER) is a fundamental task in natural language processing (NLP). However, it often has difficulty in handling entity ambiguity due to limited contextual clues. To address this challenge, traditional multimodal methods introduce social media images to assist text understanding, but the semantic deviation and noise of images seriously restrict the multimodal modeling effect. This paper proposes NER-S, a novel multimodal NER framework that integrates text-guided Scalable Vector Graphics (SVG) as reliable visual information and combines structural semantic consistency score (SSCS) to select images with high visual and semantic consistency as auxiliary information to improve entity recognition performance. Specifically, the original text is first input into the SVG image generation model to generate candidate images. Then, the optimal image is selected by SSCS and input into the multimodal named entity recognition model as the final visual supplement. Experiments on Twitter-2015 and Twitter-2017 datasets demonstrate the effectiveness of NER-S, with F1 scores of 76.60% and 86.87%, respectively. Our model outperforms all text-only baselines and exhibits comparable or superior robustness and generalization capabilities to existing multimodal models with real-world images.