<p>Navigating unmanned ground vehicle (UGV) through off-road environments is critical for various tasks like exploration and rescue. Unlike scenarios allowing offline global planning based on prior knowledge, online navigation becomes essential due to the dynamic nature of these tasks. Although deep reinforcement learning (DRL) offers promise for mapless autonomous navigation due to its end-to-end advantages, existing approaches often rely solely on goal positions. This neglects the complex distribution of obstacles along the path, leading to inefficient interactions with the environment during training. To address this challenge, a deep reinforcement learning framework is proposed for autonomous navigation guided by wayshowers. Initially, a new metric is developed based on multilevel analysis to generate elevation maps, aiding in the identification of optimal wayshowers. Upon integrating wayshower information with other inputs, a multi-head attention (MHA) module is incorporated into DRL network, which includes a length attention mechanism to enhance focus on recent historical observation sequences to promote model convergence. Furthermore, the reward function is reshaped to offer dense reward signals, thereby resolving the sparse reward problem inherent in goal-driven methods. To validate the proposed approach, experiments are conducted on several off-road maps in both the Carla and Gazebo simulators. The results demonstrate the superiority of our method not only in simple environments but also in more challenging scenarios.</p>