Path planning is the basis for realizing intelligent Blue Army combat simulation and military intelligence, and traditional reinforcement learning algorithms have slow convergence or difficult convergence in solving path planning. This paper proposes an experience grading strategy and combines it with heuristic reward function and TD3 (Twin Delayed Deep Deterministic policy gradient) algorithm to form a path planning algorithm model with heuristic experience grading TD3. The experience grading strategy will enhance the learning of important experiences generated during the training process, which can not only reduce the number of invalid training times in the pre-training period to accelerate the convergence of the algorithm, but also can make the algorithm model converge more stably. The performance of the proposed algorithm is verified through a series of experiments in a simulation environment.