Multi-robot patrolling is a key feature for various applications related to surveillance and security, and it has been studied from several different perspectives, ranging from techniques that devise optimal off-line strategies to implemented systems. This paper differs from other literature that seeks theoretically optimal patrol strategies. Instead, it focuses on multi-robot formation patrolling and online planning of patrol paths, drawing inspiration from the three-three principle in military tactics. Firstly, a three-vehicle cooperative patrol structure framework based on a leader and followers was constructed using reinforcement learning. Secondly, key technologies such as the reward function for autonomous cooperative formation and formation expansion were designed. The performance of the Adaptive Exploration Rate and ε-greedy strategies, as well as the Q-learning and Sarsa algorithms, were studied in a two-dimensional simulation environment within the context of this paper. Finally, the feasibility and effectiveness of the designed three-vehicle formation and its extended formation patrolling were verified through simulations in the Webots environment. This provides a research foundation for multi-autonomous intelligent unmanned cluster formation patrolling, which is also the content of our future research.