Reinforcement learning (RL) is an important class of machine learning techniques, in which intelligent agents optimize their behavior by observing and evaluating the outcomes of their repeated interactions with their environment. A key to successfully engineering such agents is to provide them with the opportunity to engage in a large number of such interactions safely and at a low cost. This is often achieved through developing simulators of such interactions, in which the agents can be trained while also different training strategies and parameters are explored. However, specifying and implementing such simulators can be a complex endeavor requiring a systematic process for capturing and analyzing both the goals and actions of the agents and the characteristics of the target environment. We propose a framework for model-driven goal-oriented development of RL simulation environments. The framework utilizes a set of extensions to a standard goal modeling notation that allows concise modeling of a large number of ways by which an intelligent agent can interact with its environment. Though subsequent formalization, the model is used by a specially constructed simulation engine to simulate agent behavior, such that off-the-shelf RL algorithms can use it as a training environment. We present the extension of the goal modeling language, sketch its semantics, and show how models built with it can be made executable.