With the development of sensing technologies, intelligent robots are usually equipped with both microphones and cameras. Therefore, it is possible for the robot to perceive and understand the physical world from both the visual and audio information. Especially in practical scenarios, it often happens that visual and audio information appear in the video streaming simultaneously and some objects are making sound, e.g., instrument playing, people talking, etc. In this chapter, we propose a novel task of sound-indicated active object detection, which leverages the audio information to actively detect the sounding object in the video, and a two-stream weakly supervised deep learning architecture is designed to solve this task accordingly. Case studies are conducted on both the dataset and realistic scenarios demonstrating that the proposed method is able to actively detect the sounding object effectively.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Sound-Indicated Active Object Detection

  • Di Guo,
  • Huaping Liu

摘要

With the development of sensing technologies, intelligent robots are usually equipped with both microphones and cameras. Therefore, it is possible for the robot to perceive and understand the physical world from both the visual and audio information. Especially in practical scenarios, it often happens that visual and audio information appear in the video streaming simultaneously and some objects are making sound, e.g., instrument playing, people talking, etc. In this chapter, we propose a novel task of sound-indicated active object detection, which leverages the audio information to actively detect the sounding object in the video, and a two-stream weakly supervised deep learning architecture is designed to solve this task accordingly. Case studies are conducted on both the dataset and realistic scenarios demonstrating that the proposed method is able to actively detect the sounding object effectively.