Vision Transformers (ViTs) have demonstrated exceptional performance across various computer vision tasks. ViTs utilize the attention mechanism for feature extraction, thereby enhancing global information capture compared to convolutional neural networks. Currently, the adversarial attack methods designed for ViTs focus on creating adversarial samples using the degree of attention between tokens, while neglecting the impact of ViT’s nonlinear factors. In this paper, we propose to Skip Attention Gradient (SAG) to avoid the nonlinear effects of attention by leveraging the linear nature of ViTs to generate adversarial examples. We demonstrate that the adversarial patch selected by SAG can promote a more linear propagation of adversarial effects, thereby enhancing the attack’s effectiveness. Our method achieves a robust accuracy on DeiT-B that is 7% lower compared to the Patch-Fool method.