Scholarlink

<p>Video saliency detection models have been achieving steady, significant improvements thanks to rapid advances in deep learning and the wide availability of large-scale training sets. However, deep learning-based visual-audio fixation prediction is still in its infancy. At present, only a few visual-audio sequences have been furnished, with real fixations being recorded in real visual-audio environments. Hence, it would neither be efficient nor necessary to recollect real fixations under the same visual-audio circumstances. To address this problem, this paper promotes a novel weakly supervised approach that alleviates the demand for large-scale training sets for visual-audio model training. By using only the video category tags, we propose the selective class activation mapping (SCAM) and its upgrade (SCAM+). In the spatial-temporal-audio circumstance, the former follows a coarse-to-fine strategy to select the most discriminative regions, which are usually capable of exhibiting high consistency with real human-eye fixations. The latter equips the SCAM with an additional multigranularity perception mechanism, making the whole process more consistent with that of the real human visual-audio system. Moreover, we distill knowledge from these regions to obtain completely new spatial-temporal-audio (STA) fixation prediction (FP) networks, enabling broad applications when video tags are unavailable. Without resorting to any real human-eye fixation, the performances of these STAFP networks are comparable to those of fully supervised networks. The code and results are publicly available at <a href="https://github.com/guotaowang/STANet">https://github.com/guotaowang/STANet</a>.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Weakly supervised visual-auditory fixation prediction with multigranularity perception

  • 期刊论文

  • 发表日期：

    2026年1月16日

查看全文
  • Guotao Wang,
  • Chenglizhao Chen,
  • Deng-Ping Fan,
  • Aimin Hao,
  • Qinping Zhao

摘要

Video saliency detection models have been achieving steady, significant improvements thanks to rapid advances in deep learning and the wide availability of large-scale training sets. However, deep learning-based visual-audio fixation prediction is still in its infancy. At present, only a few visual-audio sequences have been furnished, with real fixations being recorded in real visual-audio environments. Hence, it would neither be efficient nor necessary to recollect real fixations under the same visual-audio circumstances. To address this problem, this paper promotes a novel weakly supervised approach that alleviates the demand for large-scale training sets for visual-audio model training. By using only the video category tags, we propose the selective class activation mapping (SCAM) and its upgrade (SCAM+). In the spatial-temporal-audio circumstance, the former follows a coarse-to-fine strategy to select the most discriminative regions, which are usually capable of exhibiting high consistency with real human-eye fixations. The latter equips the SCAM with an additional multigranularity perception mechanism, making the whole process more consistent with that of the real human visual-audio system. Moreover, we distill knowledge from these regions to obtain completely new spatial-temporal-audio (STA) fixation prediction (FP) networks, enabling broad applications when video tags are unavailable. Without resorting to any real human-eye fixation, the performances of these STAFP networks are comparable to those of fully supervised networks. The code and results are publicly available at https://github.com/guotaowang/STANet.