Do You Trust Your Model Explanations? An Analysis of XAI Performance Under Dataset Shift

Machine learning (ML) models are increasingly deployed in many critical settings such as medicine, where ensuring their trustworthiness is a priority. The inherent complexity and opacity of many classifiers, as well as the potential unreliability of their predictions, are contributing factors that hinder the use of Artificial Intelligence systems in clinical practice. In response, Explainability methods (XAI), such as AraucanaXAI, provide a human-interpretable explanation of the reasoning process behind the predicted outputs. However, the concordance between the prediction of the classifier and the one produced by the surrogate explainer model (i.e. fidelity) is not always guaranteed, especially if the sample is near the decision boundary or in a region of the feature space far from the training distribution (dataset shift), where the ML predictions may be unreliable. Reliability methods assess potential ML misclassification errors: we exploit a method able to assess reliability based on the local performance of the classifier and the distance of the sample from the training set, in order to identify unreliable predictions. We then check the robustness of AraucanaXAI on unreliable predictions on a simulated scenario of Myelodysplastic Syndromes risk assessment. We show that fidelity considerably drops when AraucanaXAI is used to explain predictions flagged as unreliable. Drawing from this, we provide guidance for XAI hyperparameters tuning. With this work, we study the interplay between reliability and explainability, crucial requirements for Trustworthy AI. Code is available at https://github.com/bmi-labmedinfo/araucana-xai/tree/master/notebooks .