Visual Transformers Meet Convolutional Neural Networks: Providing Context for Convolution Layers in Semantic Segmentation of Remote Sensing Photovoltaic Imaging

The energy potential of Photovoltaic Panel (PVPs) decreases due to obstructions. Identification of rooftops and relevant objects to optimize PVPs capacity is usually done using aerial imagery. However, the manual process is tedious, costly, and time-consuming involving subjective expert judgment. While machine learning-based segmentation algorithms, e.g., Convolutional Neural Network (CNNs), show high accuracy while segmenting images with extensive training data, performance decreases in the context of PVPs, due to lack of labelled data for panels and obstacles. This data scarcity results from costs of acquiring and annotating high-resolution aerial imagery implying non-robust pattern learning. This paper introduces the application of Visual Transformer (ViTs) to segment relevant objects for identifying PVPs potential using aerial images. ViTs addresses the need for models with inductive bias, allowing direct image learning rather than feature representations. Consequently, ViT requires less training data to achieve optimal performance, with advantages in scenarios with limited training data available. In our study, we compare segmentation capabilities of a ViT and a CNNs using same small size, low resolution aerial images as training data. Our results suggest better performance of ViTs compared to CNNs with the same training and test dataset for the PVP context, and slightly better results when using a hybrid CNN-ViT model. In this last case, it is clear that adding global information learned by ViTs to local information learned by CNNs yields better performance using a small amount of low-cost, low-resolution reduced data.