Numerous machine learning and deep learning models are available to determine whether a given text contains hate speech or not. These machine learning and deep learning algorithms widely help us in filtering texts, messages, tweets, and comments in social media, but most of these models lack explainability. This paper aims at building explainable ML models for hate speech detection. For these models, a twitter dataset with thousands of tweets was used to train different models. First, the tweets in dataset were preprocessed, that is, the tweets were cleaned, and using these preprocessed tweets, a logistic regression model, SVM model, and decision tree classifier were trained. These models resulted in accuracies of 89.7%, 64.9%, and 89.8%, respectively, and a LSTM model was also trained resulting in accuracy of 96.3%. Then, Shapley additive explanations (SHAP) were implemented on these three models (logistic regression, SVM, and decision tree classifier) to explain the output predicted by each model, that is, the models were made explainable. Summary plots for all the three models were plotted to know the effect of each feature on the output predicted (hate or not hate) by the model. This helps in knowing which feature or more precisely which part of input text is affecting the output prediction when compared to others. This helps in assessing the accuracy and correctness of each model that was trained from the same dataset and hence has improved the training and predicting process.