The abstract should briefly summarize the contents of the paper in The growing presence of multimedia data in contemporary digital spaces has posed a high demand of artificial intelligence systems that can comprehend information that is presented in a multi-modal form of vision, audio and language. The human perception offers a viable and naturalistically based paradigm to solve this problem because the brain has a natural habit of integrating heterogeneous information of sensory signals based on hierarchical processing, selective attention and contextual learning. Our proposed bio-inspired multimodal deep learning framework is intended to be used in cognitive multimedia perception in this paper. In particular, we propose a cross-modal attention mechanism based on biology which dynamically models the reliability of the modality and removes the noisy or missing senses. The given method uses modality-specific deep encoders and attention-based fusion strategy based on the human multisensory integration, which allows to adaptively weight visual, auditory and textual information depending on its relevance to the current situation. Extensive simulations on representative multimodal benchmarks show that the proposed framework always beats unimodal frameworks and traditional multimodal fusion schemes with respect to accuracy, robustness and interpretability. According to its experimental findings, the maximum absolute accuracy increase compared to its traditional fusion approaches is 4.4% under noisy modality conditions, with the attention mechanism also offering explainable information about modality contributions. The above findings indicate that the integration of biological and cognitive concepts with multimodal deep learning architectures results in better, more efficient, and more human-like multimedia perception systems that can be applied to practice.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

A Bio-Inspired Cross-Modal Attention Framework for Robust Multimodal Multimedia Perception

  • Kimia Shirini,
  • Sina Samadi Gharehveran,
  • Saman Rajebi,
  • Siamak Pedrammehr,
  • Roohallah Alizadehsani,
  • Juan Manuel Gorriz Saez

摘要

The abstract should briefly summarize the contents of the paper in The growing presence of multimedia data in contemporary digital spaces has posed a high demand of artificial intelligence systems that can comprehend information that is presented in a multi-modal form of vision, audio and language. The human perception offers a viable and naturalistically based paradigm to solve this problem because the brain has a natural habit of integrating heterogeneous information of sensory signals based on hierarchical processing, selective attention and contextual learning. Our proposed bio-inspired multimodal deep learning framework is intended to be used in cognitive multimedia perception in this paper. In particular, we propose a cross-modal attention mechanism based on biology which dynamically models the reliability of the modality and removes the noisy or missing senses. The given method uses modality-specific deep encoders and attention-based fusion strategy based on the human multisensory integration, which allows to adaptively weight visual, auditory and textual information depending on its relevance to the current situation. Extensive simulations on representative multimodal benchmarks show that the proposed framework always beats unimodal frameworks and traditional multimodal fusion schemes with respect to accuracy, robustness and interpretability. According to its experimental findings, the maximum absolute accuracy increase compared to its traditional fusion approaches is 4.4% under noisy modality conditions, with the attention mechanism also offering explainable information about modality contributions. The above findings indicate that the integration of biological and cognitive concepts with multimodal deep learning architectures results in better, more efficient, and more human-like multimedia perception systems that can be applied to practice.