This paper describes a method for creating human action sentences from video information, which aims to infer the purpose of human actions from video information. To develop a flexible human-robot communication method that enables robots to infer their own roles from the appearance of people, it is necessary to observe and convert human behavior into data. Therefore, we have created sentences describing human behavior by combining an action recognition model and an object detection algorithm, and we have organized the spatial relationships between objects and people. In Verification 1, a simple one-sentence explanation of the behavior was provided based on the distance between the two-dimensional (2D) coordinates of the person and the object taking the action from the video image information. Through the verification, we have identified issues in the generation of behavioral sentences. In Verification 2, we spatially organized their relationship based on the change in the three-dimensional (3D) coordinates of the object and the person. Through this validation, we were able to obtain spatial information on human behavior and object access records. In the future, by integrating these methods, we aim to generate an in-spatial action log that links human behavior and spatial information.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Preliminary Experiments for Behavior Log Generation Considering Spatial Relationship Between Actions and Objects

  • Masae Yokota,
  • Sarthak Pathak,
  • Mihoko Niitsuma,
  • Kazunori Umeda

摘要

This paper describes a method for creating human action sentences from video information, which aims to infer the purpose of human actions from video information. To develop a flexible human-robot communication method that enables robots to infer their own roles from the appearance of people, it is necessary to observe and convert human behavior into data. Therefore, we have created sentences describing human behavior by combining an action recognition model and an object detection algorithm, and we have organized the spatial relationships between objects and people. In Verification 1, a simple one-sentence explanation of the behavior was provided based on the distance between the two-dimensional (2D) coordinates of the person and the object taking the action from the video image information. Through the verification, we have identified issues in the generation of behavioral sentences. In Verification 2, we spatially organized their relationship based on the change in the three-dimensional (3D) coordinates of the object and the person. Through this validation, we were able to obtain spatial information on human behavior and object access records. In the future, by integrating these methods, we aim to generate an in-spatial action log that links human behavior and spatial information.