Large Language Models (LLMs) have revolutionized natural language processing, powering applications from virtual assistants to content generation. Despite their impressive capabilities, these models remain susceptible to jailbreak attacks through adversarial prompts. While previous research has primarily focused on the security implications of jailbreak prompts, the factual accuracy and real-world applicability of these outputs remain underexplored. This paper introduces and utilizes the JailFact-Bench dataset to investigate the factual precision and toxicity of responses generated by jailbreak prompts as compared to semantically similar, non-adversarial factuality prompts. Employing this rigorously curated dataset, we analyze the responses to five prompt-based manipulation attacks: low-resource, Base64 encoding, Caesar cipher, CipherChat, and ASCII art attack. Our comparative analysis reveals that although jailbreak prompts can circumvent safety filters, they frequently lead to outputs with substantial semantic drift and elevated risks of factual inaccuracies. These findings challenge the prevailing assumption that jailbreak outputs are primarily harmful due to their toxicity, underscoring the necessity for semantic and factual integrity in evaluating the impact of adversarial attacks on LLMs. The study emphasizes the need for robust alignment techniques that ensure the safety and reliability of LLM outputs, advocating for comprehensive mitigation strategies that address not only explicit toxicity but also factual and semantic consistency.