Deep neural networks (DNNs) for road sign classification have shown high accuracy. Despite being crucial in autonomous driving, this task is vulnerable to risks from adversarial attacks. Various methods can induce misclassification in physical settings, with scenarios like dirty signs or sticker obstructions enhancing this risk. Shadows on road signs are, however, the most frequent cause of errors. To counter such effects, adversarial training—incorporating adversarial examples into the training set—is often used. Yet, this can unintentionally lower performance on clean images by introducing adversarial features absent in clean data. Our study seeks to enhance accuracy against adversarial examples without affecting clean image performance by boosting training data diversity. We achieved this by creating a dataset from road sign illustrations. By overlaying these illustrations with varied tones and proportions on randomly selected backgrounds, and applying changes such as brightness adjustments, motion blur, and noise, we augmented the dataset. Models trained on this data showed improved generalization. Our experiments achieved over 85% accuracy on a 43-class task without traditional training data. Additionally, using just 1/20th of the original training data, our models matched the accuracy of those trained with complete datasets, reducing data collection efforts.