Due to the integration of more and more intermittent renewable energy sources, electrical Transmission System Operator (TSOs) embrace a more dynamic grid management, and zonal controllers have emerged as critical components. These controllers, powered by optimization algorithms, oversee specific zones within the power grid, intervening on the power grid to mitigate line overflows. However, their choice of actions and effectiveness depends on predefined target plans provided by human operators, necessitating decision-support tools to streamline this process. Because of reinforcement learning (RL) performance in sequential problems, RL holds promise in devising such plans, yet its reliance on extensive training iterations presents a time-consuming challenge. In this paper, we propose an emulator for zonal controllers, facilitating sufficient training iterations within reasonable computation times. Our methodology combines RL with heuristic techniques tailored to alleviate common challenges encountered in RL applications, such as reducing the action space and incorporating expert knowledge where beneficial. We further introduce a framework for modeling this hybrid approach within a specific Markov Decision Process tailored to the training phase. To systematically tackle the complexities inherent in this problem, we adopt a phased approach, progressively identifying and resolving key challenges. We obtained an emulator that can handle dangerous real-time situations while following the target plan when possible. By integrating RL with domain-specific heuristics and leveraging a structured problem decomposition strategy, our methodology offers a promising avenue for efficiently training decision-support systems for zonal controllers in power grid management.