<p>Object detection is one of the most fundamental problems to tackle in the computer vision research area. Recent advances in multimodal data streams and deep learning architectures have prompted a fast growth in the field of multimodal learning, which brings several advantages over single-modality approaches for object detection, such as improved accuracy, robustness to noise and ambiguity, handling of complex scenarios and adaptability to diverse data. Some of the biggest challenges when implementing a multimodal learning approach are the selection of the fusion strategy, design of processing architecture, modality alignment/synchronization and interpretability of such high-dimensional representations. To address this challenge, we propose a feature-level fusion architecture for object detection based on extracting YOLO features from images, spectral and rhythm features from sound using Mel-frequency cepstral coefficients, and general descriptors from radar modalities that, after timestamp and homography transformation matrix alignment, are combined with an attention mechanism into a single classification network. Preliminary experiments indicate that the proposed architecture can constitute itself as a base pipeline for several different multimodal object detection tasks in real-world applications.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Multimodal object detection: an architecture using feature-level fusion and deep learning

  • Rui Silva,
  • Eduardo Coelho,
  • Nuno Pimenta,
  • Dalila Durães,
  • Victor Alves,
  • Lourenço Bandeira,
  • José Machado,
  • Paulo Novais,
  • Pedro Melo-Pinto

摘要

Object detection is one of the most fundamental problems to tackle in the computer vision research area. Recent advances in multimodal data streams and deep learning architectures have prompted a fast growth in the field of multimodal learning, which brings several advantages over single-modality approaches for object detection, such as improved accuracy, robustness to noise and ambiguity, handling of complex scenarios and adaptability to diverse data. Some of the biggest challenges when implementing a multimodal learning approach are the selection of the fusion strategy, design of processing architecture, modality alignment/synchronization and interpretability of such high-dimensional representations. To address this challenge, we propose a feature-level fusion architecture for object detection based on extracting YOLO features from images, spectral and rhythm features from sound using Mel-frequency cepstral coefficients, and general descriptors from radar modalities that, after timestamp and homography transformation matrix alignment, are combined with an attention mechanism into a single classification network. Preliminary experiments indicate that the proposed architecture can constitute itself as a base pipeline for several different multimodal object detection tasks in real-world applications.