Bias Reduced Methods to Q-learning

It is well known that Q-learning (QL) suffers from overestimation bias, which is caused by using the maximum action value to approximate the maximum expected action value. To solve overestimation issue, overestimation property of Q-learning is well studied theoretically and practically. In general, most work on reducing overestimation bias is to find different estimators to replace maximum estimator in order to mitigate the effect of overestimation bias. These works have achieved some improvement on Q-learning. In this work, we still focus on overestimation bias reduced methods. In these methods, we focus on M samples action values, and one of these samples estimated by remaining samples’ maximum actions. We select median and max members from these new samples which are estimated by maximum actions. We call these max and median members as Bias Reduced Max Q-learning (BRMQL) and Bias Reduced Median Q-learning (BRMeQL). We first theoretically prove that BRMQL and BRMeQL suffer from underestimation bias and analyze the effect of number of M Q-functions on the performance of our algorithms. Then we evaluate the BRMQL and BRMeQL on benchmark game environments. At last, we show that BRMQL, and BRMeQL less underestimate the Q-value than Double Q-learning (DQL) and perform better than several other algorithms on some benchmark game environments.