Diversified Experience Replay for Multi-agent Reinforcement Learning

To enhance the capability of off-policy multi-agent reinforcement learning (MARL), previous research has extensively investigated the agents’ decision-making and credit assignment. However, the role of experience replay is largely overlooked, with related works limited to prioritizing samples based on their TD-errors. Despite their improvements, more accurate Q-value estimations do not guarantee better decisions, as the relative advantage of different situations is more crucial for the greedy policy. To this end, we propose Diversified Experience Replay (DivER), which increases the experience diversity in the sampled mini-batch. Agents are guided to learn behaviors that transition to more superior situations, thus enhancing training efficiency. DivER is compatible with any off-policy MARL methods and has been experimentally proven to be effective across various tasks and algorithms.