<p>Multimodal knowledge graphs (MKGs) serve as critical infrastructure for data mining by integrating structured entities with heterogeneous data, such as text and images. However, real-world MKGs frequently grapple with data sparsity, modality imbalance, and irrelevant cross-modal noise, resulting in optimization instability and poor generalization in MKG completion tasks. To mitigate these challenges, we propose entropy-regularized multimodal fusion (ERMF), a framework designed to achieve robust and explainable representation learning. Specifically, ERMF constructs a unified heterogeneous graph and employs a graph attention network (GAT) to dynamically model context-aware dependencies among structured, textual, and visual features. A context-aware gating network further recalibrates modality contributions, adaptively emphasizing informative signals while suppressing noisy ones. Crucially, to prevent overconfident or biased fusion, we introduce an entropy-regularized term that explicitly enforces a balanced modality distribution, thereby enhancing both robustness and interpretability. Extensive experiments on two benchmark datasets (FB15K-237-IMG and WN18-IMG) demonstrate that ERMF consistently outperforms state-of-the-art models, achieving substantial gains of +2.7% Hits@10 on FB15K-237-IMG and +2.57% Hits@3 on WN18-IMG. Comprehensive empirical evaluations, covering ablation studies, robustness checks, and sensitivity analyses, validate the framework’s stability and reveal that ViT-based encoders yield stronger visual semantics than VGG16, while a single-layer GAT strikes an optimal balance between expressiveness and computational efficiency. ERMF provides a principled, interpretable, and robust solution for multimodal knowledge graph completion under noisy and sparse conditions.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Entropy-regularized multimodal fusion for robust and explainable knowledge graph completion

  • Yongfang Li,
  • Chunhua Zhu,
  • Xuemin Wang,
  • Yuhong Zhang,
  • Zhihua Liu

摘要

Multimodal knowledge graphs (MKGs) serve as critical infrastructure for data mining by integrating structured entities with heterogeneous data, such as text and images. However, real-world MKGs frequently grapple with data sparsity, modality imbalance, and irrelevant cross-modal noise, resulting in optimization instability and poor generalization in MKG completion tasks. To mitigate these challenges, we propose entropy-regularized multimodal fusion (ERMF), a framework designed to achieve robust and explainable representation learning. Specifically, ERMF constructs a unified heterogeneous graph and employs a graph attention network (GAT) to dynamically model context-aware dependencies among structured, textual, and visual features. A context-aware gating network further recalibrates modality contributions, adaptively emphasizing informative signals while suppressing noisy ones. Crucially, to prevent overconfident or biased fusion, we introduce an entropy-regularized term that explicitly enforces a balanced modality distribution, thereby enhancing both robustness and interpretability. Extensive experiments on two benchmark datasets (FB15K-237-IMG and WN18-IMG) demonstrate that ERMF consistently outperforms state-of-the-art models, achieving substantial gains of +2.7% Hits@10 on FB15K-237-IMG and +2.57% Hits@3 on WN18-IMG. Comprehensive empirical evaluations, covering ablation studies, robustness checks, and sensitivity analyses, validate the framework’s stability and reveal that ViT-based encoders yield stronger visual semantics than VGG16, while a single-layer GAT strikes an optimal balance between expressiveness and computational efficiency. ERMF provides a principled, interpretable, and robust solution for multimodal knowledge graph completion under noisy and sparse conditions.