Integrated Discounted Future Prediction as Auxiliary Task for A3C

In reinforcement learning the main task of an agent is to maximize cumulative reward sum. Recently there were proposed a lot of different auxiliary tasks to be solved in parallel with the main one. This often helps to decrease the number of samples needed to train an agent and to obtain more solid environment representations. However, these representations are usually hard to interpret and researchers can evaluate agent's understanding of the environment only indirectly, using loss functions and metrics. In this work we propose a novel auxiliary task based on predicting the discounted sum of future states. We show that this task can help to make training of an A3C algorithm more sample efficient on Atari gym Pong task and make it more stable while providing a way of representations interpretation.