As the capabilities and technologies of Unmanned Aerial Vehicles (UAVs) improve, new ways of utilizing them are being investigated. We investigate the use of reinforcement learning to navigate a UAV in an unknown environment, where the room layout is initially unknown. We present a novel approach for exploring and controlling a UAV, which must locate points of interest in such rooms. In this approach, we use reinforcement learning in an online fashion, meaning that the learning is performed multiple times as our knowledge of the room improves. We present the implementation of a stochastic model predictive control approach paired with Q-learning and partition refinement, using Uppaal Stratego to synthesize near-optimal strategies for UAVs to explore, map, and locate objects in environments with no prior knowledge. To ensure the safety of those strategies, we add a pre-shield during learning and employ a post-shield on the proposed actions to be executed. We evaluate our approach using simulation and compare it against a greedy approach, in which the UAV always visits the nearest unexplored part of the map. Our evaluation shows that the approach explores all points of interest approximately 11% faster than the baseline, while also reducing the number of times a new plan must be synthesized by 33%.