Trajectory planning methods for self-directed management of Unmanned Aerial Vehicle (UAV) swarms are increasingly sought after due to their significant benefits in diverse applications, including search and rescue, surveillance, disaster response, and environmental monitoring. However, optimizing the trajectories of multiple UAVs in a dynamic environment remains a significant challenge due to the high-dimensional space and coordination complexity. Additionally, energy consumption is higher if their flight paths are not optimized. To improve the efficiency of system, a joint optimization problem is designed, including energy allocation and trajectory design. To address the challenge, the paper proposes a reinforcement learning approach for trajectory optimization of UAV swarms using a deep neural network framework with strategic power allocation to reduce energy consumption. The simulation outcomes indicate that the proposed OMA-RL model surpasses standard reinforcement learning approaches, achieving a reward score of 1975 and requiring fewer UAV movements in uncertain environments.