Penetration testing is a widely recognized method for assessing the security defense capabilities of computer network systems, traditionally conducted by human experts. Recently, automated penetration testing has emerged as an innovative approach aimed at achieving autonomous penetration testing through the use of reinforcement learning agents. However, this approach encounters significant challenges due to the high complexity of penetration testing scenarios and the inherently low sample efficiency of reinforcement learning methodologies. To address these challenges, we propose a hierarchical deep reinforcement learning model that incorporates a large language model to constrain the action space, enhancing both learning efficiency and overall performance. Firstly, we divided automated penetration testing into two levels of tasks and assigned them to two separate agents, thereby simplifying the action space for each agent. Subsequently, a large language model was introduced to further constrain the action space and leverage the prior knowledge embedded within the model to improve the agent’s exploration efficiency. To evaluate the effectiveness and reliability of the proposed approach, we constructed an experimental scenario using scanning information collected from real vulnerable environments. The experimental results demonstrate that agents utilizing our approach can learn optimal strategies more efficiently and exhibit better adaptability in environments with a large number of target hosts.