To reveal the autonomous emergence mechanism of cooperative behavior in complex systems, this study integrates reinforcement learning with evolutionary game theory to construct a Q-learning-based multi-agent system for the spatial Prisoner’s Dilemma. The concept of spatiotemporal cell is proposed as a microscopic analysis unit to systematically explore the laws governing the emergence, growth, and extinction of cooperation, while quantifying the regulatory effects of parameters through an \(\alpha -\gamma \) phase diagram. Our results show that the emergence of cooperation in the spatial Prisoner’s Dilemma system does not require external interventions as in traditional models. Instead, it relies on the autonomous exploration-belief update-strategy coordination process of agents. Specifically, only when agents within a spatiotemporal cell undergo long-term interactive learning with their neighbors and are triggered by synchronous cooperative-state pulses will they transition from the ground-state belief mode to the excited states. Macroscopically, cooperative behavior exhibits a stable two-periodic oscillation mode. The oscillation amplitude increases with the learning rate \(\alpha \) and decreases with the discount factor \(\gamma \) , and a high \(\gamma \) (valuing future rewards) conversely inhibits cooperation. We develop a spatiotemporal cell theoretical framework. Based on this theory, four phases of the system’s collective decision-making behaviors are identified in the \(\alpha -\gamma \) phase diagram: Ground state phase, Cooperative phase, Cooperation growth phase, and Cooperation extinction phase, which are highly consistent with the results of simulation experiments. This study provides a new paradigm and theoretical support for understanding the behavior emergence in ecological and social systems, as well as for the design of engineering systems such as unmanned swarms.
