Malware programs now pose a serious danger to our privacy and security since they seek to do a variety of things, including stealing our private information and taking down our systems. In the past few years, driven by data artificial intelligence techniques, such as machine learning (ML) and deep learning (DL) approaches, have shown promise in detecting malware by leveraging its behavior in terms of API calls. This contrasts the ineffective and time-consuming nature of traditional signature-based malware detection methods or statistical analysis. Malware programs now pose a serious danger to our privacy and security since they seek to do a variety of things, including stealing our private information and taking down our systems. This article includes an in-depth examination of the several XAI algorithms used in malware analysis. Additionally, we have covered the traits, difficulties, and specifications in malware analysis that conventional XAI techniques cannot satisfy the performance of KNN, Decision Tree (DT), and Multi-Layer Perceptron (MLP) across Accuracy, MCC, and F1-score. KNN and DT consistently perform well, with high accuracy and F1-scores close to 99.93%, and strong MCC values indicating reliable predictions.