Aiming at the problems of slow convergence speed and suboptimal paths encountered in path planning for Unmanned Surface Vessel (USV) using reinforcement learning algorithm, an improved path planning algorithm is proposed. In this paper, the required data from electronic charts are extracted and rasterized, converting them into raster maps. This approach ensures that simulation results align with the actual scenarios, thereby improving the accuracy of path planning. Furthermore, an improved Dyan-Sarsa algorithm is proposed, replacing the greedy algorithm with annealing algorithm for action selection. This modification aims to improve the convergence and avert local optimal solution. Building upon this, the Artificial Potential Field (APF) is introduced to adjust the reward value near the target point, thereby enhancing the algorithm’s convergence. Compared with the original Dyna-Sarsa algorithm in the same environment, simulation results show that the improved algorithm not only enhances convergence speed but also escapes local optimal solutions, ensuring that the identified paths optimize both safety and energy consumption.