The integration of Large Language Models (LLM) into automatic speech recognition (ASR) system has brought remarkable improvements, making speech a fundamental interaction modality in various communication scenarios. However, challenges remain in overcoming the difficulty of huge variability in native speakers’ pronunciation and the lack of high-quality annotated dialect data, which hinder the performance of dialect recognition. Therefore, we propose a dynamic advanced reinforcement learning on Group Relative Policy Optimization (GRPO) for robust low-resource dialect ASR task. Moreover, the proposed strategy further employing curriculum learning from easy-to-hard during reinforcement learning phase significantly outperforms supervised fine-tuning (SFT) with fully randomized training data. Ablation experiments conducted on 50-hour corpus per dialect indicating the effectiveness of the proposed method for stable RL training.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Curriculum Reinforcement Learning for Robust Low-Resource Chinese Dialect Speech Recognition

  • Leijing Hou,
  • Yingying Gao,
  • Min Liu,
  • Huihui Li,
  • Xiqing Zhang,
  • Nannan Wang,
  • Shilei Zhang

摘要

The integration of Large Language Models (LLM) into automatic speech recognition (ASR) system has brought remarkable improvements, making speech a fundamental interaction modality in various communication scenarios. However, challenges remain in overcoming the difficulty of huge variability in native speakers’ pronunciation and the lack of high-quality annotated dialect data, which hinder the performance of dialect recognition. Therefore, we propose a dynamic advanced reinforcement learning on Group Relative Policy Optimization (GRPO) for robust low-resource dialect ASR task. Moreover, the proposed strategy further employing curriculum learning from easy-to-hard during reinforcement learning phase significantly outperforms supervised fine-tuning (SFT) with fully randomized training data. Ablation experiments conducted on 50-hour corpus per dialect indicating the effectiveness of the proposed method for stable RL training.