To address the limitations of effectively integrating textual and visual information while maintaining their unique characteristics, this paper proposes a Cross-modal Attention and Contrastive Learning Neural Network (CACLN) model for multimodal fake news detection. This model employs a cross-modal attention mechanism to capture semantic associations between images and text, while incorporating contrastive learning to enhance the consistency of representation across modalities. Specifically, the CACLN model leverages Vision Transformer (ViT) to capture semantic features from images, and employs a GPT decoder to generate an image caption. The BERT is utilized for embedding of news articles and image captions, and ResNet50 is employed to capture visual features of images. The CACLN model achieves effective multimodal information fusion via cross-modal attention and employs a softmax classifier for fake news prediction. Extensive experiments on three datasets demonstrate that the CACLN model significantly gains better results than baseline models. Ablation studies reveal that caption embeddings and contrastive learning provide critical semantic information, while the cross-modal attention effectively promotes semantic alignment and feature interaction across modalities.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Multimodal Fake News Detection Based on Cross-Modal Attention and Contrastive Learning

  • Zhihua Yan,
  • Xijin Tang,
  • Zhenpeng Li

摘要

To address the limitations of effectively integrating textual and visual information while maintaining their unique characteristics, this paper proposes a Cross-modal Attention and Contrastive Learning Neural Network (CACLN) model for multimodal fake news detection. This model employs a cross-modal attention mechanism to capture semantic associations between images and text, while incorporating contrastive learning to enhance the consistency of representation across modalities. Specifically, the CACLN model leverages Vision Transformer (ViT) to capture semantic features from images, and employs a GPT decoder to generate an image caption. The BERT is utilized for embedding of news articles and image captions, and ResNet50 is employed to capture visual features of images. The CACLN model achieves effective multimodal information fusion via cross-modal attention and employs a softmax classifier for fake news prediction. Extensive experiments on three datasets demonstrate that the CACLN model significantly gains better results than baseline models. Ablation studies reveal that caption embeddings and contrastive learning provide critical semantic information, while the cross-modal attention effectively promotes semantic alignment and feature interaction across modalities.