Convolutional Neural Networks excel in various applications but remain susceptible to adversarial attacks. Even tiny alterations, such as a single-pixel shift, could drastically mislead cutting-edge state-of-the-art models. In this study, we explore this vulnerability—the adversarial example problem—attributing it primarily to limited training samples. Such a case leads to overfitting and deviation from optimal models. To overcome this challenge, we propose integrating a variety of symmetry-invariant operations into CNN designs. This strategy maximizes the use of available training data, amplifies the neural network’s expressive capacity, and empowers its robustness. Our experiments demonstrate the effectiveness of this approach against random perturbations in test data while concurrently enhancing their generalization capabilities. Overall, by augmenting CNN architectures with symmetry-invariant layers, we strive to mitigate vulnerabilities, enhancing both robustness and generalization adaptability.