Scholarlink

<p>This paper introduces DOAMRL, a novel meta-reinforcement learning (meta-RL) method that extends the Model-Agnostic Meta-Learning (MAML) framework. The method addresses a key limitation of existing meta-RL approaches, which struggle to effectively use suboptimal demonstrations to guide training in sparse reward environments. DOAMRL effectively combines reinforcement learning (RL) and imitation learning (IL) within the inner loop of the MAML framework, with dynamically adjusted weights applied to the IL component. This enables the method to leverage the exploration strengths of RL and the efficiency benefits of IL at different stages of training. Additionally, DOAMRL introduces a meta-learned parameter offset, which enhances targeted exploration in sparse reward settings, helping to guide the meta-policy toward regions with non-zero rewards. To further mitigate the impact of suboptimal demonstration data on meta-training, we propose a novel demonstration data enhancement module that iteratively improves the quality of the demonstrations. We provide a comprehensive analysis of the proposed method, justifying its design choices. A comprehensive comparison with existing methods in various stages (including training and adaptation), using both optimal and suboptimal demonstrations, along with results from ablation and sensitivity analysis, demonstrates that DOAMRL outperforms existing approaches in performance, applicability, and robustness.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Demonstration and offset augmented meta reinforcement learning with sparse rewards

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2025年2月28日

查看全文
  • Haorui Li,
  • Jiaqi Liang,
  • Xiaoxuan Wang,
  • Chengzhi Jiang,
  • Linjing Li,
  • Daniel Zeng

摘要

This paper introduces DOAMRL, a novel meta-reinforcement learning (meta-RL) method that extends the Model-Agnostic Meta-Learning (MAML) framework. The method addresses a key limitation of existing meta-RL approaches, which struggle to effectively use suboptimal demonstrations to guide training in sparse reward environments. DOAMRL effectively combines reinforcement learning (RL) and imitation learning (IL) within the inner loop of the MAML framework, with dynamically adjusted weights applied to the IL component. This enables the method to leverage the exploration strengths of RL and the efficiency benefits of IL at different stages of training. Additionally, DOAMRL introduces a meta-learned parameter offset, which enhances targeted exploration in sparse reward settings, helping to guide the meta-policy toward regions with non-zero rewards. To further mitigate the impact of suboptimal demonstration data on meta-training, we propose a novel demonstration data enhancement module that iteratively improves the quality of the demonstrations. We provide a comprehensive analysis of the proposed method, justifying its design choices. A comprehensive comparison with existing methods in various stages (including training and adaptation), using both optimal and suboptimal demonstrations, along with results from ablation and sensitivity analysis, demonstrates that DOAMRL outperforms existing approaches in performance, applicability, and robustness.