Power grids have become valuable targets for both physical and cyber-attacks. A huge corpus of research and engineering best practices exist with regard to threat modeling. As robustness alone is considered inadequate to counter today’s attack vectors, resilience has become the focus of research. In this vein, learning agents, e. g., based on deep reinforcement learning, have shown a lot of promise both in deriving attack vectors and providing resilience against those. Pure model-free approaches would start training from scratch and need to re-discover already known attacks. Offline reinforcement learning provides a suitable alternative to this. However, the generation of an offline training dataset that reflects the threat model at hand while still providing an extensive dataset to train on is a research gap. Moreover, power grid simulations usually feature time series, which introduce stationarity issues that could invalidate an agent’s policy. In this paper, we apply state machines as tools to generate data based on a well-known attack. We further show how to verify the impact of the environment’s non-stationarity.