Large-scale UAV swarm confrontation based on hierarchical attention actor-critic algorithm

In large-scale unmanned aerial vehicle (UAV) swarm confrontation scenarios, the design of decision-making and coordination strategies becomes extremely difficult. Multi-Agent Reinforcement Learning (MARL), as a novel decision-making approach to address this issue, faces challenges such as poor scalability and the curse of dimensionality. To overcome these challenges, the paper proposes a Hierarchical Attention Actor-Critic (HAAC) algorithm. The HAAC algorithm includes a centralized critic network based on a Hierarchical Two-stage Attention Network (H2ANet), along with a hierarchical actor policy network that combines rules and reinforcement learning approaches. H2ANet is specifically designed to model the relationships between UAVs and extract crucial information from neighboring UAVs, enabling the generation of advanced cooperative and competitive strategies. The HAAC algorithm effectively reduces the dimensionality of both action and state spaces. Experimental results conducted demonstrate that the HAAC algorithm outperforms existing methods and is able to extend its learned policies to large-scale scenarios.