Evolutionary Reinforcement learning (ERL) algorithms effectively address the sample efficiency problem in continuous state-action spaces by integrating gradient-based reinforcement learning (RL) with gradient-free evolutionary algorithms (EAs). However, traditional ERL methods still face limitations, including insufficient processing of the state space in collected samples, diversity-limited fitness evaluation based solely on rewards leading to reduced sample diversity, and differences in EA-RL sample distributions that hinder the value function learning. This paper proposes Evolutionary Reinforcement Learning - Causal Decoupling Representation (ERL-CDR). which addresses the issue of insufficient state space processing by introducing a causal representation learning mechanism that disentangles task-irrelevant features from decision-critical features. A behavioral diversity mechanism is used to evaluate the population, promoting more comprehensive exploration of the sample space and increasing sample diversity. Finally, a novel value function is introduced to better represent the sample space of policy populations, which, together with the RL value function, effectively guides policy learning. Experimental results show that this method significantly improves convergence speed, stability, and final performance in complex continuous control tasks. These methods are of research significance for complex continuous reinforcement learning tasks.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

ERL-CDR: Evolutionary Reinforcement Learning with Causal Decoupling Representation

  • Zuohao Wu,
  • Guang Li,
  • Zihao li,
  • Jiaqi Wei,
  • Mingsheng Shang

摘要

Evolutionary Reinforcement learning (ERL) algorithms effectively address the sample efficiency problem in continuous state-action spaces by integrating gradient-based reinforcement learning (RL) with gradient-free evolutionary algorithms (EAs). However, traditional ERL methods still face limitations, including insufficient processing of the state space in collected samples, diversity-limited fitness evaluation based solely on rewards leading to reduced sample diversity, and differences in EA-RL sample distributions that hinder the value function learning. This paper proposes Evolutionary Reinforcement Learning - Causal Decoupling Representation (ERL-CDR). which addresses the issue of insufficient state space processing by introducing a causal representation learning mechanism that disentangles task-irrelevant features from decision-critical features. A behavioral diversity mechanism is used to evaluate the population, promoting more comprehensive exploration of the sample space and increasing sample diversity. Finally, a novel value function is introduced to better represent the sample space of policy populations, which, together with the RL value function, effectively guides policy learning. Experimental results show that this method significantly improves convergence speed, stability, and final performance in complex continuous control tasks. These methods are of research significance for complex continuous reinforcement learning tasks.