Scholarlink

<p>Multimodal Relation extraction (MMRE) aims to identify the relationship between two entities in text containing visual cues. Visual content is crucial for MMRE, but existing methods have shortcomings in processing fine correlations between different modalities, and it is difficult to capture key visual information. To address this problem, we propose the<b> B</b>ilateral<b> G</b>ated<b> F</b>usion with<b> P</b>refix-<b>T</b>uning (BGF-PT) framework, which aims to deeply understand the connections between text, entity pairs, and visual elements (e.g., images or objects). First, we employ a prompt-based autoregressive encoding to construct a feature interaction network using entity-specific prefixes and object-oriented prefixes as Bridges to enhance the relevance between elements. In addition, the BGF-PT also designs a bilateral gated fusion module, which can accurately distinguish the importance of image and target information. Furthermore, we introduce a generative decoder for entity type constraints, which effectively narrows the search range of candidate relations by taking entity types as prior knowledge. Extensive verification on benchmark datasets shows excellent performance advantages of the BGF-PT framework, which fully proves its effectiveness and advancement in multi-modal relation extraction tasks.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Encoder-decoder with bilateral gated fusion for multimodal relation extraction

  • 期刊论文

  • 发表日期：

    2025年5月3日

查看全文
  • Chunyu Lu,
  • Tianran Chen,
  • Duo Shang,
  • Jun Luo,
  • Xin Hui,
  • Ruhui Shi

摘要

Multimodal Relation extraction (MMRE) aims to identify the relationship between two entities in text containing visual cues. Visual content is crucial for MMRE, but existing methods have shortcomings in processing fine correlations between different modalities, and it is difficult to capture key visual information. To address this problem, we propose the Bilateral Gated Fusion with Prefix-Tuning (BGF-PT) framework, which aims to deeply understand the connections between text, entity pairs, and visual elements (e.g., images or objects). First, we employ a prompt-based autoregressive encoding to construct a feature interaction network using entity-specific prefixes and object-oriented prefixes as Bridges to enhance the relevance between elements. In addition, the BGF-PT also designs a bilateral gated fusion module, which can accurately distinguish the importance of image and target information. Furthermore, we introduce a generative decoder for entity type constraints, which effectively narrows the search range of candidate relations by taking entity types as prior knowledge. Extensive verification on benchmark datasets shows excellent performance advantages of the BGF-PT framework, which fully proves its effectiveness and advancement in multi-modal relation extraction tasks.