We consider the problem of worst-case assurance for black-box perimeter defense policies, such as those produced by machine learning, in discrete environments. We provide two assurance methodologies from game theory that can extract optimal attacker inputs against a specified defender policy. Our contributions are as follows. First, we model the defense scenario as an extensive-form game that allows us to solve optimally for a security level, but only over small time horizons due to a high computational overhead. We then show how the scenario can be modeled as a single-player stochastic game (a Markov decision process) and solved using a Q-learning algorithm for reinforcement learning. This approach allows for policies to be computed over much longer time horizons, at the cost of an optimality guarantee. We then validate both of our approaches on a simple linear defense environment and provide visualizations of the extracted security levels for two representative test policies.