<p>Efficient navigation in dynamic environments remains a critical challenge for autonomous robots in industrial manufacturing, search-and-rescue operations, and automated warehousing. Traditional path-planning algorithms struggle to adapt to real-time obstacle movements, while conventional reinforcement learning (RL) approaches cannot model long-range spatial dependencies. This paper presents Transformer-Enhanced Deep Q-Learning (Transformer-DQN), a novel framework that integrates transformer architectures with Deep Q-Networks (DQN) to address these limitations. The model dynamically captures obstacle interactions by integrating multi-head self-attention mechanisms and Cartesian positional encoding and optimizes navigation in cluttered environments. The Transformer’s self-attention mechanism to solve specific, well-known limitations of DQN in robotic path planning, namely the capture of long-range spatial dependencies and dynamic obstacle relationships in a non-Markovian setting. Hyperparameter tuning via Optuna ensures a balance between exploration and exploitation, while prioritized experience replay enhances training stability. Experimental results demonstrate significant advancements over baseline methods: 20<InlineEquation ID="IEq1"> <EquationSource Format="TEX">\(\times\)</EquationSource> </InlineEquation>20 Grids: Reduces average pathfinding time by 33.65% (209.5s <InlineEquation ID="IEq2"> <EquationSource Format="TEX">\(\rightarrow\)</EquationSource> </InlineEquation> 139s) and collisions by 37.5% compared to vanilla DQN. 30<InlineEquation ID="IEq3"> <EquationSource Format="TEX">\(\times\)</EquationSource> </InlineEquation>30 Grids: Achieves a 94.6% time reduction (9125s <InlineEquation ID="IEq4"> <EquationSource Format="TEX">\(\rightarrow\)</EquationSource> </InlineEquation> 492.5s) and 33.3% fewer collisions, showcasing superior scalability. Adaptive Performance: Outperforms PPO (70% vs. 85% success rate) and classical planners (RRT/D) in dynamic settings, approaching optimal path lengths (28 vs. 25 steps). The Transformer-DQN’s ability to generalize across grid sizes and dynamically replan in real-time positions it as a robust solution for time-sensitive applications. Theoretical analysis confirms convergence guarantees, while empirical validation demonstrates substantial improvements in both time optimization and traversal efficiency and reduced computational overhead. This work provides a foundational step towards bridging the gap between simulated and real-world robotic systems by establishing a robust learning framework in a dynamically controlled environment. Offering a scalable framework for autonomous navigation in unstructured, dynamic environments.</p>

Transformer-enhanced deep Q-Learning for adaptive robot path planning in dynamic environments

期刊论文
发表日期：

2026年6月1日

查看全文

Harish Sharma,
Ritu Tiwari,
Shubham Shukla,
Sushant Kumar

摘要

Efficient navigation in dynamic environments remains a critical challenge for autonomous robots in industrial manufacturing, search-and-rescue operations, and automated warehousing. Traditional path-planning algorithms struggle to adapt to real-time obstacle movements, while conventional reinforcement learning (RL) approaches cannot model long-range spatial dependencies. This paper presents Transformer-Enhanced Deep Q-Learning (Transformer-DQN), a novel framework that integrates transformer architectures with Deep Q-Networks (DQN) to address these limitations. The model dynamically captures obstacle interactions by integrating multi-head self-attention mechanisms and Cartesian positional encoding and optimizes navigation in cluttered environments. The Transformer’s self-attention mechanism to solve specific, well-known limitations of DQN in robotic path planning, namely the capture of long-range spatial dependencies and dynamic obstacle relationships in a non-Markovian setting. Hyperparameter tuning via Optuna ensures a balance between exploration and exploitation, while prioritized experience replay enhances training stability. Experimental results demonstrate significant advancements over baseline methods: 20 \(\times\) 20 Grids: Reduces average pathfinding time by 33.65% (209.5s \(\rightarrow\) 139s) and collisions by 37.5% compared to vanilla DQN. 30 \(\times\) 30 Grids: Achieves a 94.6% time reduction (9125s \(\rightarrow\) 492.5s) and 33.3% fewer collisions, showcasing superior scalability. Adaptive Performance: Outperforms PPO (70% vs. 85% success rate) and classical planners (RRT*/D*) in dynamic settings, approaching optimal path lengths (28 vs. 25 steps). The Transformer-DQN’s ability to generalize across grid sizes and dynamically replan in real-time positions it as a robust solution for time-sensitive applications. Theoretical analysis confirms convergence guarantees, while empirical validation demonstrates substantial improvements in both time optimization and traversal efficiency and reduced computational overhead. This work provides a foundational step towards bridging the gap between simulated and real-world robotic systems by establishing a robust learning framework in a dynamically controlled environment. Offering a scalable framework for autonomous navigation in unstructured, dynamic environments.