Combining Policy Gradient and Q-Learning

So far in this book, in the context of deep learning combined with reinforcement learning, Chapters 6 and 7 explained deep Q-learning with its variants. You looked at policy gradients in Chapter 8 . Neural network training requires multiple iterations, and Q-learning, an off-policy approach, enables you to reuse sample transitions multiple times, giving you sample efficiency. However, Q-learning can be unstable at times. Further, it is an indirect way of learning. Instead of learning an optimal policy directly, you first learn q-values and then use these action values to learn optimal behavior. Chapter 8 looked at the approach of learning a policy directly, giving you much better improvement guarantees. However, all the policy learning algorithms in Chapter 7 were on-policy. You used a policy to interact with the environment and made updates to the policy weights to increase the probability of good trajectories/actions while reducing the probability of bad ones. After each update the policy is changed, making the previous sample rollouts useless. Therefore, you had to discard the old ones and collect new trajectory rollouts after each update, making the process an on-policy leaning.