Improved Exploration Strategy for Q-Learning Based Multipath Routing in SDN Networks

Software-Defined Networking (SDN) is characterized by a high level of programmability and offers a rich set of capabilities for network management operations. Network intelligence is centralized in the controller, which is responsible for updating the routing policies according to the applications’ requirements. To further enhance such capabilities, the controller has to be endowed with intelligence by integrating Artificial Intelligence (AI) tools in order to provide the controller the ability to autonomously reconfigure the network in a timely way. In this paper, we address the deployment of a Q-learning algorithm for the routing optimization problem in terms of latency minimization. Using a direct modeling approach of the multi-path flow-routing problem, we delve deeper into the impact of the exploration-exploitation strategies on the algorithm’s performance. Furthermore, we propose a couple of improvements to the Q-Learning algorithm to enhance its performance within the considered environment. On the one hand, we integrate a congestion-avoidance mechanism in the exploration phase, which leads to effective improvements in the algorithm’s performance with regard to average latency, convergence time, and computation time. On the other hand, we propose to implement a novel strategy based on the Max-Boltzman Exploration method (MBE), which is a combination of the traditional \(\varepsilon\) $\epsilon$ - greedy and softmax strategies. The results show that, for an appropriate tuning of the hyperparameters, the MBE strategy combined with the congestion-avoidance mechanism performs better than the \(\varepsilon\) $\epsilon$ -greedy, \(\varepsilon\) $\epsilon$ -decay, and Softmax strategies in terms of average latency, convergence time, and computation time.