Recent advances in deep reinforcement learning (DRL) have significantly improved quadruped locomotion in complex environments. Traditional blind policies, relying only on proprioception, are robust and transferable but lack environmental awareness, reducing effectiveness on unseen terrains. Vision-based approaches provide foresight but often suffer from poor modality fusion, inefficient feature use, and sensitivity to noise. To address these issues, we propose a multimodal control framework that integrates visual and proprioceptive inputs, combining the strengths of both approaches. A novel gated fusion module enables adaptive, fine-grained interaction between modalities, dynamically adjusting fusion based on the current state to reduce noise and improve robustness. Benchmark results across diverse terrains show our method consistently outperforms baselines in effectiveness and generalization.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Gated Multi-modal Fusion for Vision-Based Locomotion Control of Quadruped Robots

  • Jiajun Dong,
  • Yanbin Xu,
  • Feng Dong

摘要

Recent advances in deep reinforcement learning (DRL) have significantly improved quadruped locomotion in complex environments. Traditional blind policies, relying only on proprioception, are robust and transferable but lack environmental awareness, reducing effectiveness on unseen terrains. Vision-based approaches provide foresight but often suffer from poor modality fusion, inefficient feature use, and sensitivity to noise. To address these issues, we propose a multimodal control framework that integrates visual and proprioceptive inputs, combining the strengths of both approaches. A novel gated fusion module enables adaptive, fine-grained interaction between modalities, dynamically adjusting fusion based on the current state to reduce noise and improve robustness. Benchmark results across diverse terrains show our method consistently outperforms baselines in effectiveness and generalization.