Reinforcement Learning (RL) algorithms have succeeded in several challenging domains. Classic Online RL job schedulers can learn efficient scheduling strategies but often take tens of thousands of timesteps to explore the environment and adapt to a randomly initialized policy. Current RL schedulers overlook the importance of learning from pre-recorded datasets and improving upon existing customized heuristic policies. Data-driven RL (a.k.a batch RL) presents the prospect of policy optimization from pre-recorded datasets without online environment interaction. We explore two data-driven RL methods: Behaviour Cloning and Offline RL, which aim to learn policies from pre-recorded data without interacting with the environment. These methods address the challenges concerning the cost of data collection and safety, particularly pertinent to real-world applications of RL. Although the data-driven RL methods produce good results, we show that the performance is highly dependent on the quality of the pre-recorded datasets used during training. We demonstrate that by effectively incorporating prior expert demonstrations to pre-train the RL scheduling agent, we can short-circuit the random exploration phase to learn a reasonable policy with minimal online training. We utilize batch RL as a launchpad to learn effective scheduling policies from datasets collected using an Oracle or custom heuristic policies. We demonstrate that by combining offline pre-training and minimal online RL training, we can achieve comparable performance and reduce the training time by \(\sim 3x\) compared to the state-of-the-art online RL method. This framework is highly effective for pre-training using prior datasets with batch RL methods and well suited to continuous improvement using online learning.