To address the inefficiency and convergence challenges faced by Deep Q-Network (DQN) and its variants in discovering attack paths for penetration testing, this paper proposes a Dual-layer Double-agent Deep Reinforcement Learning Algorithm (D3PC). D3PC combines the strengths of improved DQN algorithms and Proximal Policy Optimization (PPO). The proposed method enhances the original DQN by introducing advantage function normalization and a dual-network architecture to mitigate overestimation issues. Furthermore, it incorporates Generalized Advantage Estimation (GAE) and gradient clipping techniques from PPO-Clip to facilitate hierarchical training. Initially, the security analysis platform MulVAL is employed to convert network topology and vulnerability information into an attack graph. Subsequently, the D3PC algorithm trains agents to continuously explore and update their strategies within the attack graph, ultimately identifying the attack path with the highest cumulative reward. Experimental results across various scenarios demonstrate that, compared to the popular PPO and DQN algorithms, D3PC converges more efficiently and accurately to the optimal attack path, thereby significantly improving the effectiveness of penetration testing.