Current vision-language pretraining (VLP) models exhibit significant vulnerability to adversarial examples. Compared to white-box attacks, transfer attacks on unseen models better reflect real-world scenarios and offer greater research value. However, existing methods generally fail to effectively disrupt vision-language semantic alignment mechanism, limiting the transferability of adversarial examples. To address this issue, we propose DiSCo, a unified adversarial example generation framework. Specifically, we propose a perturbation generation strategy that disrupts semantic topological structures and image–text feature similarity, introducing cross-modal inconsistency that degrades downstream task performance. To mitigate the influence of redundant feature dimensions, we project image-text features into a semantic subspace constructed via clustering and hierarchical sampling before optimizing the adversarial trajectory. Furthermore, we design a cross-modal attention-based perturbation modulation mechanism that identifies text-relevant image regions and adjusts perturbations accordingly. Extensive experiments across various downstream tasks and VLP architectures demonstrate that our method significantly outperforms state-of-the-art approaches, with superior transferability and generalization.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

DiSCo: Disrupting Semantic Consistency for Transferable Cross-Modal Adversarial Attacks

  • Peirou Liang,
  • Meng Yang,
  • Zhiqian Wu,
  • Peng Yuan Zhou,
  • Yong Liao

摘要

Current vision-language pretraining (VLP) models exhibit significant vulnerability to adversarial examples. Compared to white-box attacks, transfer attacks on unseen models better reflect real-world scenarios and offer greater research value. However, existing methods generally fail to effectively disrupt vision-language semantic alignment mechanism, limiting the transferability of adversarial examples. To address this issue, we propose DiSCo, a unified adversarial example generation framework. Specifically, we propose a perturbation generation strategy that disrupts semantic topological structures and image–text feature similarity, introducing cross-modal inconsistency that degrades downstream task performance. To mitigate the influence of redundant feature dimensions, we project image-text features into a semantic subspace constructed via clustering and hierarchical sampling before optimizing the adversarial trajectory. Furthermore, we design a cross-modal attention-based perturbation modulation mechanism that identifies text-relevant image regions and adjusts perturbations accordingly. Extensive experiments across various downstream tasks and VLP architectures demonstrate that our method significantly outperforms state-of-the-art approaches, with superior transferability and generalization.