The rapid development of AI-generated content (AIGC) has heightened the threat of multimodal misinformation, posing significant security concerns. Existing misinformation detection methods have not addressed privacy concerns or have only considered the privacy of single-modal data, overlooking the semantic correlation privacy among multimodal data. In this paper, we propose a cloud-edge-end collaboration-based privacy-preserving multimodal misinformation detection (CPMD) scheme, safeguarding the correlation privacy of multimodal data while efficiently detecting misinformation. To achieve this, we introduce a local differential privacy-based multimodal Fourier dynamic protection (MFDP) scheme, thereby achieving efficient multimodal fusion and semantic correlation protection. We also design a multi-head spatiotemporal correlation attention-based large model pre-training protection (MSLP) method and DP-based Gradient Correlation Protection (DPGC) method, which ensure high-utility gradient correlation protection. Furthermore, we utilize a multimodal misinformation detection transformer (MMDT) to perform high accuracy recognition. Extensive experiments demonstrate that, compared to the HAMMER method, our CPMD method improves the accuracy of multimodal misinformation detection by 5.08%.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Towards Efficient and Secure Multimodal Misinformation Detection

  • Jianhao Wei,
  • Yixiao Li,
  • Xiaolong Li,
  • Ming Zhao,
  • Jin Zhang,
  • Limei Liu,
  • Junfeng Yang

摘要

The rapid development of AI-generated content (AIGC) has heightened the threat of multimodal misinformation, posing significant security concerns. Existing misinformation detection methods have not addressed privacy concerns or have only considered the privacy of single-modal data, overlooking the semantic correlation privacy among multimodal data. In this paper, we propose a cloud-edge-end collaboration-based privacy-preserving multimodal misinformation detection (CPMD) scheme, safeguarding the correlation privacy of multimodal data while efficiently detecting misinformation. To achieve this, we introduce a local differential privacy-based multimodal Fourier dynamic protection (MFDP) scheme, thereby achieving efficient multimodal fusion and semantic correlation protection. We also design a multi-head spatiotemporal correlation attention-based large model pre-training protection (MSLP) method and DP-based Gradient Correlation Protection (DPGC) method, which ensure high-utility gradient correlation protection. Furthermore, we utilize a multimodal misinformation detection transformer (MMDT) to perform high accuracy recognition. Extensive experiments demonstrate that, compared to the HAMMER method, our CPMD method improves the accuracy of multimodal misinformation detection by 5.08%.