Over the past years, a trade-off has been observed between the performance of machine learning models and their explainability. As much as these models have become very powerful with improved performance, due to their complexity and black box nature, it has made it difficult to use them in fields where an understanding of the decision-making process is of great importance such as in medical, military, and financial security applications. This study is set out to train five machine learning models: Logistic Regression, SVM, Random Forest, XGBoost, and LightGBM for fraud detection on digital wallets, particularly mobile money platforms. Furthermore, a comparative analysis of SHAP and Permutation Feature Importance, which are Explainable Artificial Intelligence (XAI) methods to explain how the machine learning models come up with their predictions, is applied on the best performing model. The results of this work show that the XGBoost model performs impressively well on both balanced and imbalanced dataset alike. In addition, we observed that SHAP is better able to determine the important features for model prediction in balanced datasets while PFI performs best when the dataset is imbalanced.