MATD3 with multiple heterogeneous sub-networks for multi-agent encirclement-combat task

Based on the background of a multi-agent game with limited attack and defense capabilities and communication range, a game model is established to study the encirclement-combat problem of a red agent team against its blue target agent in this paper. Under the actor-critic framework of twin delayed multi-agent deep deterministic policy gradient (MATD3) algorithm, the original MATD3 algorithm is improved according to the characteristics of the game scenario to solve the problem of agents’ number attenuation, sparse reward, and high extraction frequency of invalid experience in the original algorithm. The simulation shows that the algorithm designed in this paper has improved the convergence speed, learning efficiency, and stability compared with the original algorithm. The sub-networks make the algorithm more suitable for the game scenario where the number of agents is dynamically declining; the reward potential function and prioritized experience replay (PER) based on the importance weight improve the refinement of the difference between experiences and the utilization rate of superior experiences.