<p>Recent advancements in multimodal sarcasm detection (MSD) have made significant progress in understanding the interplay between textual and visual cues. However, existing methods tend to overemphasize cross-modal semantic alignment, consequently neglecting sarcasm cues that are independently embedded within each modality. In this paper, we present DCPNet, a novel Dual-channel Cross-modal Perception Network, to integrate unimodal and cross-modal features via the incorporation of comprehensive structural semantics. To capture rich topological relationships within each modality, we introduce a Graph Topology Extraction and Enhancement Module (GTEE) that builds graph structures from both text and image features, facilitating deeper semantic representation. Additionally, we propose a Cross-Modal Multi-Scale Feature Fusion (CMFF) module that aligns and integrates features from both text and image at multiple scales, ensuring the capture of comprehensive contextual information. An attention mechanism is incorporated to assign appropriate weights to the textual and visual features, thereby optimizing the fusion process for more accurate sarcasm detection. Extensive experiments conducted on the MMSD and MMSD2.0 benchmark datasets demonstrate that DCPNet outperforms existing state-of-the-art (SOTA) methods in both accuracy and robustness.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

DCPNet: a comprehensive framework for multimodal sarcasm detection via graph topology extraction and multi-scale feature fusion

  • Youjiang Fang,
  • Liang Zhang,
  • Shihao Wang,
  • Wenyuan Zhang,
  • Yuxin Wang,
  • Yuanyuan Liu,
  • Xiaopeng Wei,
  • Xin Yang

摘要

Recent advancements in multimodal sarcasm detection (MSD) have made significant progress in understanding the interplay between textual and visual cues. However, existing methods tend to overemphasize cross-modal semantic alignment, consequently neglecting sarcasm cues that are independently embedded within each modality. In this paper, we present DCPNet, a novel Dual-channel Cross-modal Perception Network, to integrate unimodal and cross-modal features via the incorporation of comprehensive structural semantics. To capture rich topological relationships within each modality, we introduce a Graph Topology Extraction and Enhancement Module (GTEE) that builds graph structures from both text and image features, facilitating deeper semantic representation. Additionally, we propose a Cross-Modal Multi-Scale Feature Fusion (CMFF) module that aligns and integrates features from both text and image at multiple scales, ensuring the capture of comprehensive contextual information. An attention mechanism is incorporated to assign appropriate weights to the textual and visual features, thereby optimizing the fusion process for more accurate sarcasm detection. Extensive experiments conducted on the MMSD and MMSD2.0 benchmark datasets demonstrate that DCPNet outperforms existing state-of-the-art (SOTA) methods in both accuracy and robustness.