Large Language Model (LLM) agents are increasingly used in AI-supported education. Effective inter-agent communication boosts collaborative problem-solving efficiency and lowers the cost of deploying LLM-driven educational systems. However, few studies have systematically examined how different communication strategies affect on agents’ problem-solving performance, which is essential for developing effective collaborative applications with LLM agents. Inspired by principled scenarios in learning science and engineering, our study examines four communication modes, teacher-student interaction, peer-to-peer collaboration, reciprocal peer teaching, and critical debate, in a dual-agent, chat-based mathematical problem-solving environment powered by OpenAI’s GPT-4o model. On the MATH benchmark, our results show that dual-agent setups outperform single agents, with peer-to-peer collaboration achieving the highest accuracy. Dialogue acts like statements, acknowledgment, and hints play a key role in collaborative problem-solving. These findings show that multi-agent frameworks can boost collaborative performance, but carefully crafted effective communication strategies remain essential for tackling complex tasks in AI-supported education.