Modern smart buildings operate in dynamic environments where achieving an optimal balance between energy efficiency and occupant comfort is complicated by partial observability, stochastic occupancy patterns, delayed actuator responses, and the coupled dynamics of thermal, air quality, and lighting systems. This growing complexity has motivated the increasing adoption of data-driven control methods, in which learning algorithms can complement or even surpass traditional model-based strategies. Reinforcement learning (RL) provides a model-free approach capable of deriving control policies directly from data. However, many RL architectures rely on recurrent neural networks (RNNs), such as long short-term memory (LSTM) networks, which often struggle to capture long-term dependencies due to hidden state compression. Transformer-based agents, such as the Deep Transformer Q-Network (DTQN), mitigate part of this limitation by using self-attention to selectively focus on relevant historical information. Nevertheless, their performance may degrade in highly non-stationary environments if temporal focus is not adapted dynamically. To address this issue, we propose an Adaptive Deep Transformer Q-Network (Adaptive DTQN) for non-stationary building control tasks. Adaptive DTQN operates at the level of individual attention heads, dynamically re-weighting their contributions based on real-time statistics such as attention entropy and token-level variance, thereby enabling the network to flexibly adjust its temporal focus as conditions evolve. We evaluate Adaptive DTQN in a simulated multi-variable smart building environment with hidden regime changes, realistic actuator delays, and stochastic occupancy. Results demonstrate that Adaptive DTQN consistently achieves robust performance across reward, comfort, and stability metrics, while keeping energy use within acceptable bounds, maintaining a favorable trade-off under changing temporal dependencies and underscoring its potential for real-world building automation.