With the rapid development of agent technology, multiagent systems (MASs) have become a key technology for solving complex collaborative tasks. The key challenges in MASs are which agents to coordinate and how to coordinate with these agents. Traditional multiagent reinforcement learning (MARL) methods try to distribute the global team payoffs to each agent by value decomposition, but cannot perfectly represent the relationship between the global value function and the local value function. In this paper, we make full use of the local interaction structure between agents to model the coordination relationship, and propose a structural coordination-based MARL method, which can train multiagents’ policies in a coordinated manner. By exploiting the explicit coordination graph (CG), we first propose a coordinated MARL method. The coordinated MARL models the coordination between agents as a distributed constraint optimization problem (DCOP) and employs the Shapley value technique for credit assignment. Furthermore, in the case that the CG is unknown, we propose a self-attention mechanism to dynamically construct an implicit CG, allowing agents to consider the status and action information of other relevant agents. Experimental results show that the proposed MARL based on structural coordination outperforms the state-of-the-art methods in various multiagent coordination applications.