Scholarlink

<p>Scene graph generation (SGG) is pivotal for enhancing visual understanding, yet it faces challenges such as expensive training annotations and long-tail distribution issues. Recent studies attempt to address these challenges by using a fixed language parser and an object detector to obtain triplet labels for augmenting SGG data. However, this approach reduces diverse, semantically rich captions to structured labels. In this paper, we propose a novel masked Vision-and-Language (V&amp;L) pre-training framework to improve SGG representation learning from free-form captions and image patches. We combine semantic-aligned masking strategies with multimodal masked modeling objectives to learn scene-related visual and textual representations. This framework can be further fine-tuned on SG data to achieve more accurate SGG results. Evaluations on the Visual Genome benchmark show competitive results in both the Language-Supervised and Fine-Tuned settings. Our framework raises Recall@100 from 4.1 to 8.5 under the Language-Supervised setting and outperforms most two-stage SGG models under the Fine-Tuned setting, highlighting the effectiveness of self-supervised pre-training on large-scale, noisy captioning data.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Enhancing scene graph generation via semantic-aligned masked vision-and-language pre-training

  • 期刊论文

  • 发表日期：

    2026年6月30日

查看全文
  • Xiaoguang Chang,
  • Teng Wang,
  • Lele Xu,
  • Changyin Sun

摘要

Scene graph generation (SGG) is pivotal for enhancing visual understanding, yet it faces challenges such as expensive training annotations and long-tail distribution issues. Recent studies attempt to address these challenges by using a fixed language parser and an object detector to obtain triplet labels for augmenting SGG data. However, this approach reduces diverse, semantically rich captions to structured labels. In this paper, we propose a novel masked Vision-and-Language (V&L) pre-training framework to improve SGG representation learning from free-form captions and image patches. We combine semantic-aligned masking strategies with multimodal masked modeling objectives to learn scene-related visual and textual representations. This framework can be further fine-tuned on SG data to achieve more accurate SGG results. Evaluations on the Visual Genome benchmark show competitive results in both the Language-Supervised and Fine-Tuned settings. Our framework raises Recall@100 from 4.1 to 8.5 under the Language-Supervised setting and outperforms most two-stage SGG models under the Fine-Tuned setting, highlighting the effectiveness of self-supervised pre-training on large-scale, noisy captioning data.