Coordinating and controlling robot swarms to achieve collective navigation in complex environments presents significant challenges. This paper proposes a learning-based hierarchical navigation solution for aerial swarm robotics in cluttered environments, which integrates the RRT* path planning algorithm with the D3QN reinforcement learning approach. The proposed hierarchical architecture decomposes the swarm navigation problem into two layers: global path planning and distributed local navigation. To enhance policy generalization in increasingly complex scenarios, we incorporate curriculum learning into the training process. Extensive simulation results demonstrate that our method outperforms the traditional D3QN algorithm in scalability, generalization ability, and navigation performance, effectively controlling UAV swarms to accomplish navigation tasks in challenging environments. Real-world experiments with Crazyflie quadrotors further validate the algorithm’s feasibility and robustness, confirming its effectiveness and reliability in executing swarm navigation tasks in real environments.