Large Language Models (LLMs) are increasingly being integrated into conversational agents across critical domains, ranging from customer support to task automation. While their capabilities have improved significantly, their susceptibility to adversarial inputs has raised serious concerns regarding safety and reliability. Specifically, threats such as prompt injection, jailbreaking, and training data leakage can cause LLMs to disclose sensitive information or perform unauthorized actions. In this work, we propose a supervised learning-based vulnerability scanner designed to assess the security posture of LLM-integrated chatbots by analyzing their responses. The core of our system is a multi-class text classifier built on top of transformer architectures. The classifier is trained to assign severity scores to chatbot outputs, indicating the likelihood and extent of security violations. We constructed a labeled dataset of synthetic responses generated by popular LLMs, categorized into four severity levels ranging from benign to critical. Four transformer-based models: RoBERTa, DeBERTa, ELECTRA, and DistilBERT were trained and evaluated on this dataset. Our experimental results indicate that ELECTRA achieves the best overall performance, reaching a classification accuracy of 97.08 %. The proposed system has been implemented as an end-to-end assessment tool, capable of interfacing with live chatbot APIs, executing adversarial payloads, and producing actionable security reports.