Deep Reinforcement Learning (DRL) has demonstrated exceptional accomplishments in a variety of domains, including image recognition and automation. However, its potential in logistics and transportation, particularly in addressing routing issues, remains largely untapped. Evolutionary Algorithms (EA) on the other hand, have seen widespread use in solving combinatorial optimization problems. Surprisingly, the use of EA and DRL techniques to solve combinatorial optimization problems has received little attention in the existing literature. Motivated by these research gaps, this study proposes a novel Evolutionary Reinforcement Learning (ERL) approach to solve the Traveling Salesman Problem (TSP). To improve the policy generated by a deep neural network, we use the synergy between the EA and DRL frameworks. Notably, the weights associated with the actor component are important, especially in non-policy methods. Using the power of EA, we build a population of weights and seamlessly integrate them into the DRL framework, with the goal of significantly improving TSP results. We applied two EAs, namely the Genetic Algorithm (GA) and the Red Deer Algorithm (RDA), and proposed two ERL-based approaches, the ERL-GA and the ERL-RDA, respectively. The conducted computational experimentation has shown that the ERL-RDA outperforms the ERL-GA as well as the basic DRL framework.