Despite the significant advances that Large Language Models (LLMs) offer in processing vast amounts of data and providing actionable insights quickly, their application in the technical field of cybersecurity poses significant challenges. These include the tendency to produce hallucinatory and unreliable results when these models are tested on questions where factuality is important. Furthermore, while Retrieval Augmented Generation (RAG) systems are useful in enriching model answers with relevant information, they struggle with issues related to retrieval speed, choice of embeddings and thresholds and handling multi-hop queries. This paper describes these challenges and discusses strategies to overcome them in order to improve the adaptability and reliability of these models in responding to rapidly evolving cybersecurity threats.