PTGroup: An Automated Penetration Testing Framework Using LLMs and Multiple Prompt Chains

Penetration testing is an effective means of maintaining network security. To address the challenge of high labor costs in traditional penetration testing, researchers have been investigating the potential of automated solutions. In this paper, we introduce an automated penetration testing framework with multi-agent characteristics, named PTGroup, which utilizes the prior knowledge and textual comprehension capabilities of Large Language Models (LLMs). The framework follows the operational mode of ReAct, executing a multi-round decision-making process through successive Thought-Act-Observe cycles, thereby sequentially accomplishing automated penetration testing tasks. We conducted experiments using different LLMs and demonstrated that our framework is capable of adapting to various LLMs. In addition, we proposed an approach to improve PTGroup's adaptability to different vulnerability environments by designing multiple prompt chains and we demonstrated the efficacy of this approach through a series of experiments. Finally, we propose the directions for improvement of the PTGroup, encompassing the use of more specialized LLMs and the adoption of automated prompt chain generation.