Frequent and substantial voltage fluctuations, attributed to the widespread adoption of electric vehicles, demand response initiatives, and the integration of renewable energy sources, present significant challenges to contemporary distribution grids. Electric utilities are currently experiencing major issues related to the unprecedented levels of load peaks as well as renewable penetration. This underscores the importance of voltage regulation, the essential task of ensuring that bus voltage magnitudes remain within optimal ranges, in modern distribution grids. The management of residential voltage levels primarily depended on utility-owned equipment such as load-tap-changing transformers, voltage regulators, and capacitor banks. Recently, a batch reinforcement learning (RL) framework utilizing linear function approximation was proposed in (Xu et al. 2019). Another characteristic of utility-owned equipment is their limited operational lifespan, necessitating daily or even monthly basis. Such configurations have been effective in traditional distribution grids without (or with low) renewable generation, and with slowly varying load.