Exploring the Reliability of SHAP Values in Reinforcement Learning

Explainable artificial intelligence (XAI) is an increasingly important research field, fueled by the need for reliability and accountability in applications. For reinforcement learning (RL), achieving explainability is particularly challenging because agent decisions depend on the context of a trajectory, which makes data temporal and non-i.i.d. In the field of XAI, Shapley values and SHAP in particular are among the most widely used techniques. In this work, we investigate how SHAP performs in explaining RL models, especially in multidimensional action spaces that other XAI-for-RL methods struggle with. In particular, we make three contributions: (1) We investigate how design choices of the SHAP approach affect SHAP accuracy for RL models. We investigate the size of the so-called background data that is utilized to represent absent features, as well as the selection method with which the background data is formed. We find that SHAP for RL requires only modest amounts of background data and that clustering is preferred over sampling as a selection method. (2) Additionally, we analyze how SHAP-based feature importance relates to overall agent performance (return). We find that while feature importance is often correlated to agent performance, notable exceptions occur, especially for environments that are sensitive or fragile in the sense that small changes in actions may lead to catastrophic failure. However, since a significant correlation is found in the majority of the investigated environments, SHAP proves to be a valuable XAI tool for RL with multidimensional, continuous actions. (3) Illustratively, we show the time evolution of SHAP values and caution against misinterpreting sharp changes therein.