Mapless navigation has gained significant attention in recent years, yet designing such navigation agents remains complex, requiring intricate environment and agent dynamics models. Reinforcement learning (RL) offers a robust framework for solving control problems. This paper explores mapless navigation from visual input using a data-driven, model-free, off-policy RL approach. A mobile robot, simulated in Gazebo and controlled using the Robot Operating System, is equipped with an onboard camera and LiDAR sensor. Various reward formulations were tested, and an image-based reward system leveraging a fine-tuned U-Net model trained on expert demonstrations was proposed. This formulation enhances exploration and success rates. Experimental results show that for goal-driven mapless navigation in unknown environments, the segmentation reward improves the success rate. Furthermore, the inclusion of the goal into the agent’s state provides performance improvements. In continuous action spaces, the segmentation-based reward produces fewer collisions. Lower success rates and fewer collisions hint at its effectiveness in encouraging collision-free exploration.