<p>In multimodal relation extraction (MRE), discrepancies between textual and visual representations and inadequate relation perception significantly constrain cross-modal reasoning. Furthermore, processing massive unstructured text–image pairs and constructing complex cross-modal graphs entail computationally intensive steps—such as high-dimensional embedding, dense graph reasoning, and cross-modal attention alignment. These operations demand highly efficient graph computing capabilities, representing a crucial scenario for high-performance computing (HPC). To address these challenges, we propose an MRE model introducing a synergistic graph-based reasoning framework. First, cross-modal contrastive learning aligns distinct modalities and reduces distributional disparities. Next, a relation-aware semantic fusion module bolsters mutual perception between text and relations via a tailored heterogeneous graph. Here, “relations” explicitly participate as independent nodes, dynamically modeling bidirectional influences through an improved gated-attention mechanism optimized for efficient inference and resource allocation. Additionally, to bridge the semantic gap, a knowledge-driven cross-modal enhancement module constructs explicit knowledge-inference paths between textual entities and visual objects using an external knowledge base, effectively mitigating conceptual biases and strengthening fine-grained semantic fusion. Extensive experiments on the MNRE dataset demonstrate our model’s superiority over state-of-the-art approaches across three metrics, with a notable 1.44% improvement in the F1 score. Furthermore, our model exhibits exceptional robustness even in low-resource scenarios. The related code can be found at <a href="https://github.com/djtuNLP/KGRA">https://github.com/djtuNLP/KGRA</a></p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

KGRA: A knowledge-guided and relation-aware model for enhanced multimodal relation extraction

  • Wei Zheng,
  • Guoyin Li,
  • Zhenlin Zhang,
  • Hongfei Lin

摘要

In multimodal relation extraction (MRE), discrepancies between textual and visual representations and inadequate relation perception significantly constrain cross-modal reasoning. Furthermore, processing massive unstructured text–image pairs and constructing complex cross-modal graphs entail computationally intensive steps—such as high-dimensional embedding, dense graph reasoning, and cross-modal attention alignment. These operations demand highly efficient graph computing capabilities, representing a crucial scenario for high-performance computing (HPC). To address these challenges, we propose an MRE model introducing a synergistic graph-based reasoning framework. First, cross-modal contrastive learning aligns distinct modalities and reduces distributional disparities. Next, a relation-aware semantic fusion module bolsters mutual perception between text and relations via a tailored heterogeneous graph. Here, “relations” explicitly participate as independent nodes, dynamically modeling bidirectional influences through an improved gated-attention mechanism optimized for efficient inference and resource allocation. Additionally, to bridge the semantic gap, a knowledge-driven cross-modal enhancement module constructs explicit knowledge-inference paths between textual entities and visual objects using an external knowledge base, effectively mitigating conceptual biases and strengthening fine-grained semantic fusion. Extensive experiments on the MNRE dataset demonstrate our model’s superiority over state-of-the-art approaches across three metrics, with a notable 1.44% improvement in the F1 score. Furthermore, our model exhibits exceptional robustness even in low-resource scenarios. The related code can be found at https://github.com/djtuNLP/KGRA