LEHR: LLM-Driven Evolutionary Hybrid Rewards for Multi-agent Reinforcement Learning
摘要
Reward design is pivotal in reinforcement learning (RL), particularly in environments with sparse rewards, where the quality of the reward function has a direct impact on the algorithm’s performance. This challenge becomes even more pronounced in multi-agent reinforcement learning (MARL), where balancing individual agent rewards with global rewards demands careful consideration in reward function design. To address these complexities, we propose LEHR: LLM-Driven Evolutionary Hybrid Rewards, a framework leveraging the reasoning and generation capabilities of large language models (LLMs) to develop multi-layered reward functions tailored for MARL scenarios. LLMs generate hybrid rewards that integrate global, local, and adaptive components, providing precise feedback to individual agents while promoting effective team coordination. The incorporation of a Selector module enables the LLM to generate new solutions based on selected reward functions, emulating evolutionary algorithms through selection, crossover, and mutation processes. By limiting LLMs to the reward generation phase and excluding them from the RL training process, we effectively reduce inference costs while ensuring robust reward design. Experiments in the Multi-Agent Particle Environment (MPE) reveal significant improvements in agent adaptability and collaboration, demonstrating the effectiveness of LEHR in optimizing MARL.