The Proximal Policy Optimization (PPO) algorithm is known to have issues such as extended training time and repetitive exploration, attributed to its high computational capacity and uneven sampling exploration. Therefore, this paper proposes the integration of the International Regulations for Preventing Collisions at Sea (COLREGs) as constraints within the PPO algorithm. Specifically, the global reference path obtained from the Artificial Potential Field (APF) method is utilized to construct the range field function, which in turn shapes the reward function and guides exploration direction. Additionally, APF's path nodes are inputted into the internal experience buffer and policy network of the PPO algorithm, allowing for emulation of ideal behavior patterns by USV during initial learning phases and avoidance of invalid path attempts. Finally, output paths are smoothed to enhance navigation efficiency. Simulation experiments demonstrate that the enhanced PPO algorithm accelerates exploration time per step by 47.72% compared to the unimproved version, with approximately 53.33% faster convergence steps - resulting in improved economy.