Reconsidering Stochastic Policy Gradient Methods for Traffic Signal Control

Traffic management through traffic light control is essential for reducing congestion and improving traffic flow in cities. Recent progress in deep reinforcement learning (DRL) has shown potential for enhancing traffic light efficiency. Intuitively, probabilistic policies can promote exploration in reinforcement learning and prevent it from becoming stuck in local solutions. However, the conditions under which probabilistic policies work well in actual traffic light control are not fully understood. In this study, we focused on the use of probabilistic policies within DRL and investigated the effectiveness of stochastic policies for signal control under various conditions using a traffic simulator. We scrutinized four main stochastic approaches: using a softmax function with a temperature parameter, adding noise to the intermediate layer, employing NoisyNet, and implementing Bayes by Backprop. These methods were evaluated against the reinforcement learning benchmarks for traffic signal control (RESCO) and, a set of standards for assessing traffic control systems. The results revealed that the simplest approach, that is the softmax function with appropriate temperature parameters, performed the best in tuning the temperature parameter. The findings of this study provide empirical insights into how the temperature parameter in this function correlates with different traffic conditions.