Recent developments in multi-agent reinforcement learning (MARL) have integrated game-theoretic concepts to improve coordination and learning. Common approaches including centralized training with decentralized execution (CTDE), knowledge sharing, and agent communication have been applied to algorithms such as proximal policy optimization (PPO). However, ensemble methods remain underexplored in PPO-based MARL. In this paper, we investigate the effect of an ensemble method in pure cooperative settings and propose plans-managed proximal policy optimization (PMPPO), a novel ensemble-based method that employs a hierarchical policy structure. Experimental results show that PMPPO outperforms standard CTDE and knowledge-sharing baselines in terms of average return.