Deep learning models are highly susceptible to adversarial attacks. Existing methods often operate in the RGB (Red, Green, and Blue) space, relying on perturbations of the \({L_p}\) -norm that are perceptible and limited in transferability to black-box models. Moreover, attack methods manipulating latent variables in diffusion models typically struggle to enhance attack performance against black-box models significantly or to balance attack efficacy and visual imperceptibility effectively. We propose DiffAttack-X, an effective diffusion model-based adversarial attack method to address these challenges. To enhance attack effectiveness, bi-level routing attention is introduced to reduce the correlation between adversarial samples and their corresponding correct labels. Furthermore, focal loss enhances focus on small-object features, increasing classification difficulty. For visual imperceptibility, group squeeze-and-excitation attention and self-attention mechanisms maintain structural integrity across spatial and channel dimensions, and adaptive mean squared error loss constrains semantic deviations at the pixel level, preserving visual similarity. Experiments on the ImageNet dataset show that DiffAttack-X outperforms existing methods, with performance gains of 5.4% and 10.3% across 11 black-box transfer scenarios and four adversarially trained models, along with notable improvements under two purification methods. Visualizations further validate the superiority of DiffAttack-X in attack effectiveness and stealth. Code is available at https://github.com/Achestnut00/Diffattack-X.
