Scholarlink

<p>Hand–Object Interaction Action Recognition (HOIAR) is pivotal in enhancing human–computer interaction across domains such as augmented reality (AR), virtual reality (VR), and robot Programming by Demonstration. Recognizing actions in hand–object interaction scenarios poses significant challenges due to complex motion patterns and semantic relationships. This paper introduces a multimodal framework, HOINet, which integrates 3D hand poses, 6D object poses, and RGB images to capture structural and semantic cues essential for accurate recognition. By employing feature-level fusion, HOINet identifies interaction objects, generates corresponding action verbs, and comprehends interaction semantics. Experimental evaluations on the H2O and FPHA datasets demonstrate HOINet’s superior performance, achieving Top-1 accuracies of 98.36% and 93.57%, respectively. This framework not only advances HOIAR but also offers potential applications in VR, AR, and industrial settings for intuitive interaction and robot task programming. The source code and data are available at&#xa0;<a href="https://github.com/lneverd/HOINet">https://github.com/lneverd/HOINet</a>.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Multimodal fusion for enhanced hand–object interaction recognition

  • 期刊论文

  • 发表日期：

    2026年3月27日

查看全文
  • Jie Zhang,
  • Lixin Ma,
  • Hao Zhang,
  • Xiaoheng Zhao,
  • Chunsheng Hua

摘要

Hand–Object Interaction Action Recognition (HOIAR) is pivotal in enhancing human–computer interaction across domains such as augmented reality (AR), virtual reality (VR), and robot Programming by Demonstration. Recognizing actions in hand–object interaction scenarios poses significant challenges due to complex motion patterns and semantic relationships. This paper introduces a multimodal framework, HOINet, which integrates 3D hand poses, 6D object poses, and RGB images to capture structural and semantic cues essential for accurate recognition. By employing feature-level fusion, HOINet identifies interaction objects, generates corresponding action verbs, and comprehends interaction semantics. Experimental evaluations on the H2O and FPHA datasets demonstrate HOINet’s superior performance, achieving Top-1 accuracies of 98.36% and 93.57%, respectively. This framework not only advances HOIAR but also offers potential applications in VR, AR, and industrial settings for intuitive interaction and robot task programming. The source code and data are available at https://github.com/lneverd/HOINet.