The rapid progress in Artificial Intelligence (AI) and statements by some leading figures in the industry suggest that Artificial General Intelligence (AGI) may be on the verge of becoming a reality. The timeline is highly uncertain, but once AGI emerges, it can be used to refine its own efficiency, potentially resulting in an explosive increase in capability known as a singularity. The potential for far-reaching effects demands that we consider the possible outcomes and risks. Artificial superintelligence (ASI) represents the next level of AI, at which it would surpass even the most gifted humans in cognitive competence. This paper critically examines the potential end-game implications of AGI and ASI systems, particularly the presence or absence of human-in-the-loop oversight. Building on recent advancements in large language models (LLMs) and multimodal AI, the study investigates how advanced AI might inadvertently or deliberately erode safeguards against weapons proliferation, exacerbate geopolitical tensions, and undermine global controls on high-risk technologies. Further, it offers a comprehensive taxonomy of existential risks associated with AI, including Artificial Narrow Intelligence (ANI), AGI, and ASI, culminating in the introduction of a pioneering multimodal nuclear weapon-related jailbreaking study, revealing critical vulnerabilities and potential misuse of next-generation AI systems.