Model-Agnostic Policy Explanations: Biased Sampling for Surrogate Models

Model-agnostic explanation methods can be used to interpret relative feature importance for machine learning models without the need for understanding the model’s architecture. In particular, LIME [16] provides a method for creating surrogate models, such as linear regression or decision trees, to generate local explanations describing decision boundaries and feature importance around individual points in the state space. Such explanations can be further extended to global explanations, describing the general feature importance of the model for the full state space. In reinforcement learning (RL), an agent’s policy is defined by a sequence of decisions given different situations, rather than by a single decision. In contrast to supervised learning explanation methods, explanations for models produced by RL should capture feature importance and decision boundaries in the policy space and not in terms of a local or global explanation. The current approach for XAI for RL is Policy summarization [23], typically by visual, gradient based methods. Policy summarization characterizes a path a policy would take or feature effects on a policy through a path. Visual explanations consist of feature explanations that highlight regions most effecting a policy model. In this paper, we design a novel method of sampling the state space to bias surrogate models towards policy interpretation and analyze the effects of doing so in terms of percent of the policy captured. Our method creates a surrogate model that acts as a policy summarizing method highlighting the feature importance of state values relative to an execution of the policy. We further analyze trade-offs of stability and global model decision capture by evaluating episode distance and percent uniform capture.