While fine-tuning Large Language Models (LLMs) for single-agent tasks has seen significant progress, adapting these methods to multi-agent environments presents two challenges: limited understanding of complex inter-agent relationships and inefficiency of outcome-based action evaluation. To address these challenges, we propose a framework for fine-tuning LLMs in multi-agent environments. First, we enhance base LLM agents’ understanding of agent relationships like coordination and competition using self-play data generated by advanced LLM agents. Second, we introduce a judge model, trained on contrastive action pairs collected via Monte Carlo (MC) approach, to directly evaluate action. This judge model significantly improves evaluation efficiency compared to outcome-based evaluation, enabling broader exploration and generation of high-quality preference pairs for training. We validate our method in WaterAllocation, a multi-agent environment featuring multi-round auctions. Experiments show substantial performance gains under both competitive and mixed-motive objectives, enabling agents based on small open-source models (7B/8B/13B) to outperform agents based on GPT-4. Moreover, our trained agents also exhibit robust generalization across diverse roles and modified environments.