Scholarlink

<p>Visual-Language Pre-training (VLP) Models demonstrate exceptional capability in understanding the interactions between images and text, yet they remain vulnerable to multimodal adversarial examples. Investigating adversarial attacks against VLP models, particularly in generating highly transferable multimodal adversarial examples in black-box settings, is crucial for developing more robust and practical models. Current research typically leverages modality interaction and data diversity to enhance attack transferability. However, unlike unimodal learning, VLP models heavily rely on the feature learning of both images and text to accurately align semantics across different modalities. Directly applying unimodal data augmentation methods independently on each modality may disrupt inter-modal semantic matching, thus leading the perturbations in the wrong direction. To address this challenge, we propose a joint multimodal data augmentation-based transfer attack method, termed JDA-Attack. Specifically, during the iterative generation of adversarial examples, we perform multimodal collaborative augmentation through linear interpolation of images and concatenation of corresponding text sequences. It is able to enhance data diversity while preserving the semantic relationships between image-text pairs, thus effectively improving the transferability of multimodal adversarial examples. Extensive experiments on various datasets and VLP models demonstrate that our method achieves superior attack transferability.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

JDA-attack: leveraging joint multimodal data augmentation to enhance adversarial transferability of vision-language pre-training models

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2025年9月8日

查看全文
  • Xujie Ren,
  • Caikun Zhang,
  • Guoqiang Han,
  • Bin Chen,
  • Zepeng Fan

摘要

Visual-Language Pre-training (VLP) Models demonstrate exceptional capability in understanding the interactions between images and text, yet they remain vulnerable to multimodal adversarial examples. Investigating adversarial attacks against VLP models, particularly in generating highly transferable multimodal adversarial examples in black-box settings, is crucial for developing more robust and practical models. Current research typically leverages modality interaction and data diversity to enhance attack transferability. However, unlike unimodal learning, VLP models heavily rely on the feature learning of both images and text to accurately align semantics across different modalities. Directly applying unimodal data augmentation methods independently on each modality may disrupt inter-modal semantic matching, thus leading the perturbations in the wrong direction. To address this challenge, we propose a joint multimodal data augmentation-based transfer attack method, termed JDA-Attack. Specifically, during the iterative generation of adversarial examples, we perform multimodal collaborative augmentation through linear interpolation of images and concatenation of corresponding text sequences. It is able to enhance data diversity while preserving the semantic relationships between image-text pairs, thus effectively improving the transferability of multimodal adversarial examples. Extensive experiments on various datasets and VLP models demonstrate that our method achieves superior attack transferability.