Resolving Action Delay: Multi-agent Reinforcement Learning Based on State Prediction

Action delay is very common in practical applications and can significantly impact the effectiveness of reinforcement learning, especially in multi-agent scenarios, due to the lag in actions. To address this issue, we propose a Predictive States-Based Multi-Agent reinforcement learning framework (PSMA). We introduce a predictive structure to forecast future states based on action sequences and the current state, ensuring accurate predictions even in large-delay scenarios. The decision network generates actions according to the predicted states to offset delays. We extend this predictive structure to multi-agent scenarios, achieving collaborative training between the predictive network and the decision network, addressing the issue of action delays in multi-agent systems. We provide a version based on MADDPG(Multi-Agent Deep Deterministic Policy Gradient) and validate its performance in the Multi-Agent Particle Environment. In large-delay collaborative and competitive scenarios, PSMA can converge rapidly to achieve high-performance outcomes.