This paper presents a comparative study on the application of drone-assisted infrared thermography coupled with state-of-the-art machine learning models, including Vision Transformers (ViTs) and YOLOv8, for efficient and accurate defect detection in Photovoltaic (PV) systems. The research outlines the methodology for on-site inspections and details the integration of drones to capture high-resolution thermal imagery, identifying minute anomalies indicative of potential system failures. By employing advanced image processing techniques and training AI models, the study compares the performance of these models in accurately identifying and classifying PV defects. A segmentation model based on TensorFlow was trained and used to detect the location of PV panels in the camera imagery, followed by the separate application of YOLOv8 and ViTs to classify defects in the detected PV panels. The dataset used to train the models was carefully curated and prepared specifically for this study, representing another contribution of this paper. This approach not only enhances detection capabilities but also streamlines the processes of data collection, analysis, and reporting, ultimately leading to improved decision-making for system operators and stakeholders.