Dec-POMDPs model cooperative, sequential multi-agent decision problems. They are computationally challenging, and scaling up their performance is difficult. We describe a method for solving Dec-POMDPs in the paradigm of centralized planning with distributed execution. First, we solve a team POMDP in which agent observations are common knowledge. Then, each agent uses imitation learning to try and imitate its part of the centralized policy. Unlike some previous work, the agent not only tries to imitate its behavior within the team, but also its belief state. A final offline synchronization stage improves the likelihood that agents’ policies will be well-coordinated with each other. On standard Dec-POMDP benchmarks, our method performs better than the best Dec-POMDP model-based solution method, and QMIX, a leading multi-agent RL algorithm.