Traffic simulation is needed for planning safe routes of self-driving cars and in analyzing traffic situations of a given area. Commonly supervised learning methods of vehicle, bicycle, and pedestrian traffic models have several limitations such as drifting errors and weak generalization to novel scenarios. Reinforcement learning can address these issues but it is much slower to converge due to the large state and action spaces involved in real-world traffic. To overcome this challenge, a hybrid methodology that combines supervised learning for short-term agents kinematics and reinforcement learning for long-term trajectory planning is developed in this work, and then tested on two distinct heterogeneous traffic datasets: 1) InD dataset of intersections traffic and 2) UniD dataset of shared space traffic. The results showcase the effectiveness of this method as it outperforms the supervised learning baseline models in terms of lower average displacement errors, increased success rate, and higher survival time for simulated agents. Additionally, the generalization of this approach was demonstrated by testing it on both regular intersection traffic and shared space traffic. This approach combines the benefits of supervised learning’s ability to learn complicated systems like vehicle kinematics and reinforcement learning’s potential for long-term planning in real-world traffic situations (code and video demonstrations: https://github.com/engyasin/SLRL ).