This paper investigates a challenging scene in robot daily grasping services, where the target exists obstruction and visual ambiguity. Specifically, the fine-grained visual information of the target object is obstructed, e.g., text information. Besides, there are multiple objects similar to the appearance of the target object in the scene. To tackle this issue, this paper proposes an active target location and grasping framework based on joint language-vision-action. Firstly, we take the textual label information of the target object as a clue to guide the robot to explore objects. Then, to obtain more fine-grained text information about the objects in the scene, we enable the robot actively to pick up the object for observation like humans. Consequently, the robot can acquire the target object with detailed context from multiple objects and eliminate visual ambiguity. Finally, extensive experiments are conducted in both simulation and real-world scenes to verify the effectiveness of the proposed active target location and grasping system.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Active Target Location and Grasping Based on Language-Vision-Action

  • Xinxin Zhu,
  • Jin Liu,
  • Jialong Xie,
  • Chaoqun Wang,
  • Fengyu Zhou

摘要

This paper investigates a challenging scene in robot daily grasping services, where the target exists obstruction and visual ambiguity. Specifically, the fine-grained visual information of the target object is obstructed, e.g., text information. Besides, there are multiple objects similar to the appearance of the target object in the scene. To tackle this issue, this paper proposes an active target location and grasping framework based on joint language-vision-action. Firstly, we take the textual label information of the target object as a clue to guide the robot to explore objects. Then, to obtain more fine-grained text information about the objects in the scene, we enable the robot actively to pick up the object for observation like humans. Consequently, the robot can acquire the target object with detailed context from multiple objects and eliminate visual ambiguity. Finally, extensive experiments are conducted in both simulation and real-world scenes to verify the effectiveness of the proposed active target location and grasping system.