<p>Multimodal sarcasm detection aims to determine whether conflicting semantics arise in different modalities. Existing research, primarily relying on direct interaction between image and text, limits model performance in sarcasm detection due to the difficulty in cross-modal alignment and information integration caused by semantic differences between the modalities. In this paper, we propose a progressive interaction approach. First, unlike the traditional direct interaction approach, the pre-interaction approach is adopted by bridging the image and text through attributes to reduce the semantic difference between them. Then, contrastive learning is employed to align image and text features for better synchronization of image-text semantics. Finally, sarcasm cues are captured through the interaction between image and text for detecting sarcasm. In the pre-interaction phase, we design different components for image and text respectively for their interaction with attributes. Experiments demonstrate the excellent performance of our method on a multimodal sarcasm detection task.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

A progressive interaction model for multimodal sarcasm detection

  • Yulei Zhang,
  • Guangli Zhu,
  • Yuanyuan Ding,
  • Zhongliang Wei,
  • Lei Chen,
  • Kuan-Ching Li

摘要

Multimodal sarcasm detection aims to determine whether conflicting semantics arise in different modalities. Existing research, primarily relying on direct interaction between image and text, limits model performance in sarcasm detection due to the difficulty in cross-modal alignment and information integration caused by semantic differences between the modalities. In this paper, we propose a progressive interaction approach. First, unlike the traditional direct interaction approach, the pre-interaction approach is adopted by bridging the image and text through attributes to reduce the semantic difference between them. Then, contrastive learning is employed to align image and text features for better synchronization of image-text semantics. Finally, sarcasm cues are captured through the interaction between image and text for detecting sarcasm. In the pre-interaction phase, we design different components for image and text respectively for their interaction with attributes. Experiments demonstrate the excellent performance of our method on a multimodal sarcasm detection task.