Transformers have demonstrated potential in leveraging features for two-stage human-object interaction (HOI) detection, but a considerable performance gap persists compared to one-stage methods. We attribute this discrepancy to the limited granularity in the coverage of first-stage features. In this paper, we introduce a multi-cue injected Transformer (MIT), specifically devised for two-stage HOI detection. MIT efficiently utilizes multi-granularity information, encompassing cues related to instances, bounding boxes, 3D poses, and global context. Initially, MIT associates instances, such as humans and objects, that have potential interactive relationships using bounding box cues, subsequently fusing these instances with 3D pose to derive a fused embedding for each human-object pair. These embeddings are then refined by querying on global context feature maps. Through the hierarchical integration of these diverse cues, MIT substantially enhances HOI detection performance. Extensive experiments validate MIT’s effectiveness and its superiority to state-of-the-art methods.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

MIT: Multi-cue Injected Transformer for Two-Stage HOI Detection

  • Weilong Peng,
  • Qingfeng Chen,
  • Keke Tang,
  • Zhihao Yang,
  • Meng Xing,
  • Meie Fang

摘要

Transformers have demonstrated potential in leveraging features for two-stage human-object interaction (HOI) detection, but a considerable performance gap persists compared to one-stage methods. We attribute this discrepancy to the limited granularity in the coverage of first-stage features. In this paper, we introduce a multi-cue injected Transformer (MIT), specifically devised for two-stage HOI detection. MIT efficiently utilizes multi-granularity information, encompassing cues related to instances, bounding boxes, 3D poses, and global context. Initially, MIT associates instances, such as humans and objects, that have potential interactive relationships using bounding box cues, subsequently fusing these instances with 3D pose to derive a fused embedding for each human-object pair. These embeddings are then refined by querying on global context feature maps. Through the hierarchical integration of these diverse cues, MIT substantially enhances HOI detection performance. Extensive experiments validate MIT’s effectiveness and its superiority to state-of-the-art methods.