DAGE: Dropout with Action Gradient Estimator for Continuous Control

For actor-critic algorithms in reinforcement learning, the policy update is guided by a Q function, so the quality of the policy is largely affected by the quality of the Q function. Most work has focused on how to estimate a more accurate Q function, such as using a dropout operator. However, we show that for continuous control scenarios using deterministic policy gradient algorithms, the quality of the Q function does not depend on the accuracy of the Q function itself, but on the accuracy of the action gradient of the Q function. Motivated by this observation, we propose Dropout with Action Gradient Estimator (DAGE), which aims at estimating the action gradient of the Q function accurately instead of the Q function itself. We conduct sufficient experiments on PyBullet Control Suite, and empirically show that DAGE can estimate a more accurate action gradient and achieve better performance than baselines.