In the classic Attacker-Target-Defender Differential Game (ATDDG), the penetrating unmanned aerial vehicles (UAVs) must avoid the interceptor while seeking a path to the target. Since the procedural penetration cannot meet the requirements in diversified and complex combat scenarios, this paper proposed a deep reinforcement learning algorithm called Augmented Mesh Deterministic Policy Gradient (AMDPG), which integrates multiple Q-networks based on the Actor-Critic framework, and further enhance the performance of the algorithm by incorporating techniques such as value distributional reinforcement learning, prioritized experience replay, and curiosity-driven exploration. The simulation results show that compared with the classical algorithm, the proposed algorithm has superior performance and convergence, the success rate for breaking defense reaches 70%, higher than competitors.