Scholarlink

Referring expression comprehension (REC) from a first-person perspective plays a crucial role in embodied intelligence applications such as assistive vision systems and mobile robotics. This task seeks to accurately localize target objects in egocentric visual scenes based on natural language descriptions. However, when confronted with complex visual scenes and lengthy descriptions, previous REC methods tend to focus exclusively on the primary target. They overlook the rich auxiliary semantic cues in the expression, resulting in suboptimal localization performance. To address this limitation, we propose a novel weakly supervised framework, termed Learning with Supplemental Information (LSI), that leverages unlabeled auxiliary object information in textual descriptions to enhance localization accuracy. Specifically, we utilize a pre-trained visual grounding model to generate pseudo-labels for auxiliary objects and model their visual and semantic relationships with the main referent. These pseudo-supervised signals help the model build a richer contextual understanding by aligning auxiliary and primary cues. Additionally, contrastive learning is introduced to enhance the discriminability of auxiliary information. Experimental results on the RefEgo dataset validate the effectiveness of our approach, with improvements of 2.7% in mAP@50 and 3.0% in mIoU.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Learning Supplementary Information for First-Person Perception Referring Expression Comprehension

  • 会议论文

  • 发表日期：

    2026年

查看全文
  • Zetao Du,
  • Jianhua Yang,
  • Yan Huang,
  • Liang Wang,
  • Feng Chen,
  • Zhepeng Wang

摘要

Referring expression comprehension (REC) from a first-person perspective plays a crucial role in embodied intelligence applications such as assistive vision systems and mobile robotics. This task seeks to accurately localize target objects in egocentric visual scenes based on natural language descriptions. However, when confronted with complex visual scenes and lengthy descriptions, previous REC methods tend to focus exclusively on the primary target. They overlook the rich auxiliary semantic cues in the expression, resulting in suboptimal localization performance. To address this limitation, we propose a novel weakly supervised framework, termed Learning with Supplemental Information (LSI), that leverages unlabeled auxiliary object information in textual descriptions to enhance localization accuracy. Specifically, we utilize a pre-trained visual grounding model to generate pseudo-labels for auxiliary objects and model their visual and semantic relationships with the main referent. These pseudo-supervised signals help the model build a richer contextual understanding by aligning auxiliary and primary cues. Additionally, contrastive learning is introduced to enhance the discriminability of auxiliary information. Experimental results on the RefEgo dataset validate the effectiveness of our approach, with improvements of 2.7% in mAP@50 and 3.0% in mIoU.