Deep neural networks have demonstrated remarkable effectiveness across various applications. However, their vulnerability to intelligently crafted adversarial attacks is well-documented. The literature shows that deep networks often rely on common low-level features, such as edges and textures, which are present across different object classes. For instance, classes such as cars, trucks, and ships, which belong to the broader category of automobiles, share similar characteristics. We hypothesize that these similarities make it easier for subtle adversarial noise to misclassify images into closely related classes within the feature space. Building on these insights, this research aims to enhance the robustness of deep learning algorithms by protecting classes that are likely to be misclassified. We introduce a novel data augmentation technique called Supervised Mixup, designed to retrain networks to better distinguish between closely related classes and resist various forms of subtle adversarial noise. This augmentation-based defense is attack-agnostic, making it applicable across various attacks and vulnerable pre-trained networks. Extensive experiments conducted on multiple datasets, under challenging scenarios including gray-box and white-box attacks, demonstrate the superiority of the proposed algorithm compared to recent state-of-the-art defense and data augmentation methods.