Scholarlink

<p>Memes blend images with brief text, so relying on only one of those parts makes it difficult for systems to detect hate or sarcasm. We present ReConFusion, a lightweight multimodal model that tackles this problem by linking a Sentence-BERT text encoder with a Vision Transformer image encoder through a cross-modal attention block. The attention lets the caption focus on image regions that match the words, while a small decoder tries to rebuild the original caption. This extra “caption-reconstruction” task helps the model keep fine details such as negation and ironic praise that often signal hateful intent. Tested on the Facebook Hateful Memes (FHM) and MMSD datasets, ReConFusion reaches 89.5% accuracy, 90.3% precision, 82.1% recall, 86.0% F1-score on FHM and 91.08 accuracy, 89.53 precision, 90.21 recall and 89.15% F1-score on MMSD dataset, beating strong single-modality and multimodal baselines. Ablation studies show that removing cross-modal attention decreases F1 by 0.074 on FHM and 0.099 on MMSD, while disabling caption reconstruction further reduces F1 by 0.015 on FHM and 0.013 on MMSD, confirming the importance of both components. These results demonstrate that simple multimodal fusion and auxiliary tasks like caption reconstruction can significantly improve meme content moderation and detection of nuanced content like sarcasm.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

ReConFusion: a lightweight SBERT–ViT fusion network with auxiliary caption reconstruction for detecting hateful memes with sarcastic cues

  • 期刊论文

  • 发表日期：

    2025年10月21日

查看全文
  • Hafiz Muhammad Arslan,
  • Tan Zhenhua,
  • Ayesha Sabir

摘要

Memes blend images with brief text, so relying on only one of those parts makes it difficult for systems to detect hate or sarcasm. We present ReConFusion, a lightweight multimodal model that tackles this problem by linking a Sentence-BERT text encoder with a Vision Transformer image encoder through a cross-modal attention block. The attention lets the caption focus on image regions that match the words, while a small decoder tries to rebuild the original caption. This extra “caption-reconstruction” task helps the model keep fine details such as negation and ironic praise that often signal hateful intent. Tested on the Facebook Hateful Memes (FHM) and MMSD datasets, ReConFusion reaches 89.5% accuracy, 90.3% precision, 82.1% recall, 86.0% F1-score on FHM and 91.08 accuracy, 89.53 precision, 90.21 recall and 89.15% F1-score on MMSD dataset, beating strong single-modality and multimodal baselines. Ablation studies show that removing cross-modal attention decreases F1 by 0.074 on FHM and 0.099 on MMSD, while disabling caption reconstruction further reduces F1 by 0.015 on FHM and 0.013 on MMSD, confirming the importance of both components. These results demonstrate that simple multimodal fusion and auxiliary tasks like caption reconstruction can significantly improve meme content moderation and detection of nuanced content like sarcasm.