Solar energy has been widely researched in the current research landscape given that it is one of the cleanest and most readily available sources of energy on this planet. However, the life cycle of a solar panel can be easily cut short due to the occurrences of cracks and soiling. To be able to address these faults as soon as possible, a cost-effective fault detection scheme is proposed by using a drone equipped with an RGB camera for data collection and applying YOLOv7 onto the collected data for automatic fault detection and classification. Despite the exciting prospect of this method, it suffers from low accuracy, overfitting issues and long training time compared to the use of thermal images. Therefore, this paper proposes to address these issues through data augmentation techniques and implementation of an enhanced Early Stopper. It was found that data augmentation techniques, such as brightness, exposure, noise, cutout, and blur are more effective when dust is not included in the dataset. Aside from that, it was also determined that Early Stopper using validation loss as the stopping criteria and with a patience of 10 is the most effective in tackling overfitting.