The paper addresses energy management in a building using reinforcement learning algorithms. To tackle the problem of lengthy agent training, the authors propose an approach involving an external teacher. Various methods of using an external teacher have been described, and the building energy management problem has been set out considering the premises’ occupancy, which allows the use of popular reinforcement learning algorithms with an external teacher. The results of training agents in a virtual environment show that, for all reinforcement learning algorithms, the use of an external teacher enables achieving much lower energy consumption within a fixed timeframe compared to similar algorithms without a teacher. Thus, applying reinforcement learning algorithms with an external teacher can accelerate agent training significantly and improve energy management efficiency at the initial learning stage. The proposed methods employing an external teacher can apply well-known reinforcement learning algorithms without intervening in their codes.