The increasing use of Deep Learning (DL) algorithms in cybersecurity, for example in identifying dangerous source code or Android malware, has made Artificial Intelligence (AI) and Machine Learning (ML) more crucial. AI-based cybersecurity solutions, like those in other DL application domains like Computer Vision (CV) and Natural Language Processing (NLP), are not able to clearly communicate to people the implications (which might range from detection and prediction to reasoning and decision-making). Explainable AI, or XAI, has emerged as a key area of research to address the related problems of providing human users with an interpretable or comprehensible AI model. Security operators in the cybersecurity industry are overworked due to the tens of thousands of security notifications that are received every day, the majority of which are false positives. They might be better able to recognize potential threats and have less alert weariness thanks to XAI. We do a thorough literature assessment on the relationship between cybersecurity and XAI. The security of XAI and its applications to cybersecurity (such as intrusion detection and malware classification) are especially examined from two angles in the current literature (e.g., attacks on XAI pipelines, and potential countermeasures). To characterize the security of XAI, we make use of a variety of security traits that have been researched in the literature. We list open issues that have not been resolved or have been addressed properly while also highlighting potential study opportunities.