In recent years, reinforcement learning (RL)-based online mutation strategy selection techniques have emerged as a principled learning framework for balancing exploration and exploitation in Differential Evolution. Several state-of-the-art (SOTA) DE variants have been proposed that utilize online mutation strategy selection to improve the performance of the canonical DE algorithm. This paper presents a comprehensive review of such DE variants and studies multi-armed bandit formulations for online mutation strategy selection in DE. It systematically categorizes existing DE variants based on their utilization of RL algorithms.