Buildings account for 40% of global energy consumption, with 36% attributed to heating, ventilation, and air-conditioning (HVAC) systems. Therefore, optimizing thermal system management in buildings is crucial for a sustainable future. Data-driven control strategies like deep reinforcement learning (DRL) and model predictive control (MPC) have already shown great promise, but they depend on simulation models. However, these models oversimplify real-world dynamics, leading to a “sim-to-real” gap that impairs the performance of pre-trained controllers. This paper aims to close this gap using an innovative hybrid approach. To close the sim-to-real gap, data-driven methods such as artificial neural networks (ANN) have shown promise, but unlike physics-based models, these models lack transparency, making it difficult to generalize across different HVAC systems and diagnose the sim-to-real gap. This paper proposes a hybrid approach, where data-driven models are added to a physics-based model on the level of the components, thus maintaining the physics-based model’s explainability. Given the scarcity of data on HVAC systems with degenerating components, a simulated environment is developed to replace measured data, meaning that in fact a sim-to-sim gap is closed. Using this environment, the methodology is validated for different small use cases by means of root mean squared error. It was found that the system’s parameters are not required to develop the ANNs. Finally, it was observed that the number of neurons in the first hidden layer of the model is linked to that component behaving differently, suggesting a fault detection strategy based on the ANN’s architecture.