Speech Emotion Recognition is a fundamental component of Human-Computer Interaction. Many studies in this domain rely on the extraction of higher dimensional feature sets, which lack interpretability. Therefore, in this study, a 132-dimensional feature vector has been extracted from the JL Corpus with New Zealand English. The dataset comprises of 5 primary and 5 secondary emotions. In this study, the optimal hyperparameters of Random Forest Classifier have been identified with the help of Grid Search algorithm. Furthermore, Recursive Feature Elimination has been incorporated to evaluate the optimal number of features to maximize accurate predictions. To visualize the impact of feature selection employed in our study, t-Distributed Stochastic Neighbour Embedding has been plotted before feature selection and for every feature subset. SHapley Additive exPlanations has been utilized for adding explainability to the model’s predictions and to know if the top selected features were consistent across the subsets. An accuracy of 74.17% has been obtained with 80 features, helping build insightful real-time speech emotion recognition systems.