A persistent problem in machine learning (ML) lies in the lack of transparency of the resulting predictive models. The principles of explainable artificial intelligence (XAI) are especially needed to foster trust in ML applications by opening the “black box” to understand the inner workings of ML systems. XAI methods are effective for inspecting ML models, understand how data affect their predictions, or assessing their fairness. Tools facilitate access to XAI methods, providing graphical environments for generating and analysing different types of explanations. In this work, we present a taxonomy of XAI tools, offering a detailed analysis of the methods they implement and their user-oriented features. Based on three dimensions and 31 categories, we compare 15 XAI tools, highlighting their strengths and limitations. The goals of our study are to analyse the current landscape of XAI tools, expand the use of lesser-known XAI techniques and highlight the usefulness of these tools in analysing ML models beyond performance evaluation.