Numerous organizations have adopted Artificial Intelligence (AI) in one way or another, reflecting its growing popularity in both the academic and industrial sectors. At the core of these systems lies Deep Neural Networks, a machine learning technique influenced by human neurons. Due to the increasing dependence on these systems, their autonomous nature, and the fact that in some instances they control critical infrastructures and make decisions affecting people’s health, safety, and economic disposition, a thorough investigation into their vulnerability is paramount. This study examines the vulnerability of vision backbone networks to modern adversarial attacks given numerous deep learning applications that inherit learned features of these backbone networks through finetuning and transfer learning. The vulnerability of vision backbone networks like VGG16, Resnet50, and InceptionV3 against FGSM (Fast Gradient Sign Method), PGD (Projected Gradient Descent), CW (Carlini-Wagner), DeepFool, and One-pixel attack respectively are investigated. A new measure of imperceptibility called perceptual noise is established that measures the strength of the attack method and the robustness of these backbone networks. A custom image classifier network was trained as a baseline to evaluate the robustness of the backbone networks and all models under investigation achieved average accuracies above 90%, the custom model exhibited relatively lower vulnerability to adversarial attacks. In contrast, the pre-trained models showed higher vulnerability, likely due to transfer learning. This increased vulnerability is attributed to the transferred weights from the backbone networks having distributions reasonably distant from the training dataset distribution. As a result, the pre-trained models exhibited lower perceptual noise highlighting the impacts of learned weights on network robustness during transfer learning.