Generative AI has demonstrated phenomenal potential by redefining text, image, and code creation. But despite this, such models remain vulnerable to adversarial prompts, those well-crafted inputs that are capable of biasing model output in unforeseen and malicious ways. These exposures pose important implications regarding bias, the dissemination of misinformation, and security threats, emphasizing the crucial importance of adversarial robustness in AI safety. Our work presents a new adversarial prompt generation framework that is aimed at systematically finding the vulnerabilities of generative models1. Through gradient-based perturbation methods and reinforcement learning, our method generates adversarial prompts that induce large behavioral changes in the model. We cast this problem as a constrained optimization problem, giving us a formal means to quantify and analyze AI vulnerabilities. In order to overcome these weaknesses, we introduce an adversarial training framework that leverages safety-matched loss functions and specific fine-tuning tactics in order to enhance model resilience. The aim is to make the model less vulnerable to attack while preserving its capability to create fluent and coherent content. We tested our approach on prominent Large Language Models (LLMs), such as GPT-4, Gemini, and Claude, with primary such as robustness score (RS), adversarial susceptibility index (ASI), and perturbation resistance (PR). The results of the experiment show a substantial decrease in model susceptibility, verifying the efficacy of our mitigation strategy. Though our research did not explicitly test Deepseek, our suggested framework is model-agnostic and can be used with any LLM. We are of the opinion that Deepseek shares similar vulnerabilities with our tested models, and we can modify our adversarial training approach to better improve its strength. More studies are required to determine the actual vulnerabilities of Deepseek and tweak our method to achieve the best results with this model. Our work helps the development of adversarial robustness and AI safety through the construction of a large-scale defense model against adversarial attacks. Our improvement of the resilience of a model through training on adversarials brings closer the goal of having secure, dependable, and ethically valid AI-generated data in real-life applications. In this paper, we meet the critical demand to build strong, resilient AI technologies that can fight against harmful input, fostering a responsible approach towards developing and employing generative AI technology.