This paper describes a system in which an agent can find successful action strategies in the Wumpus World—a classic environment for testing logical reasoning. Despite such successful strategies can be fully described and implemented using first-order logic, it is challenging for reinforcement learning algorithms to automatically find them because of partially observable states, sparse rewards and the logic-based nature of the problem, which characterizes the Wumpus World environment. Our solution consists of the design of sensation maps, where partial observations are accumulated, shaping the reward function, custom two-stage \(\epsilon \) -greedy action selection strategy, and curriculum learning. With these components, we were able to train agents using just a basic DQN algorithm—the pioneer of deep reinforcement learning. Our experiments confirm the good performance of the developed method improving the results described in the literature. Our solution brings reinforcement learning-based approaches closer to the complete solution of the problem designed to be solved using first-order logic.