Adversarial examples (AEs) reveal critical misalignments between machine learning models and human perception. Unlike sensitivity-based AEs (SBAEs), which introduce imperceptible perturbations to fool models, invariance-based AEs (IBAEs) modify images in ways that change human classification while keeping the model’s prediction unchanged. Both, IBAEs and SBAEs expose that models rely on spurious correlations rather than true semantic understanding. To mitigate such misbehavior, AEs must be generated and analyzed in large numbers. Generating IBAEs is a complex problem and often relies on manual processes. We propose a scalable algorithm for generating IBAEs using explainable artificial intelligence (XAI) techniques. Our method strategically replaces semantically important regions of a base image with patches from an attack image, ensuring the model retains its original classification while human classification changes. An IBAE is considered successful when a majority of humans classify it differently from its original class. We validate our approach on the Imagenette dataset and Integrated Gradients and conducted a human study with 166 participants, demonstrating its effectiveness. The results demonstrate the successful generation of IBAEs and show that IBAEs can deceive time-constrained human labelers.