Model-Based Reinforcement Learning with Hierarchical Control for Dynamic Uncertain Environments

Autonomous intelligent systems are often trained using reinforcement learning (RL), which however is difficult and inefficient in dynamic uncertain environments. Different approaches to overcome these challenges exist, including model-based RL or hierarchical control structures in factored action spaces. However, there is a lack of detailed analyses of the benefits and weaknesses of model-based RL and its combination with hierarchical control when learning action policies for such control problems. In this paper, we report the results of such an analysis. Comparing model-free and model-based RL, we show that the outputs of an internal model must be accurate with at least 50% to yield a performance gain in comparison to a model-free approach. Moreover, we explore a hierarchical control architecture that employs (model-based) control policies specialized for different environmental conditions, managed by a trained meta-agent. Our analyses of training performance indicate important directions for learning action policies in intelligent systems in dynamic uncertain environments and even for complex tasks such as in multi-tasking scenarios.