Photovoltaic (PV) systems have become a significant role player in the energy transition. However, they are subject to various external environmental factors, whether during manufacturing, installation, or throughout their lifetime. They need efficient, fast, and intelligent inspection, especially when it comes to large-scale farms. In this sense, the combination of UAV imagery with deep learning (DL) techniques will allow quality control while improving the performance of PV systems and minimizing operation and maintenance costs. The main objective of our study is to propose a deep-learning approach based on the concatenation of two DL models to automate the process of detecting and classifying PV panel anomalies from aerial Infrared thermal images. The proposed concatenation is based on the transfer learning of two CNN architectures, namely VGG19 and DenseNet201. We adopt the ‘CAVIAR’ strategy, which involves training several models in parallel and then selecting the most effective and best-performing one. We applied our approach to a set of images of real solar farms, including 20,000 images with six (06) classes of anomalies. The results show that the proposed concatenation-based model obtained the best accuracy by extracting features from two robust deep networks. The solution was able to predict the presence of anomalies with an F1 score of 86% and to correctly classify seven (07) classes with an average accuracy of 73%, providing an interesting improvement over the two models run individually.