Deep learning techniques have become increasingly important in the field of optical defect detection in recent years, often outperforming traditional image processing methods. However, the performance of deep learning methods is highly dependent on the amount of training data, which poses a challenge in industrial settings where labelled defect images are often scarce. This paper presents a novel dataset-independent approach for generating synthetic defect images that improves the performance of deep learning models while requiring minimal real-world data. Our methodology uses a pix2pix model to generate realistic defect images. The pix2pix model is trained on defect images and images where the defects are segmented out. After training, the model is applied to defect-free images to transform them into defect images using random segmentation masks. This procedure ensures that the defect location is preserved and that the synthetic data can also be used for segmentation applications. The approach is validated on an open source dataset. The results show that the proposed synthetic data generation approach reduces class imbalance and leads to improvements in model accuracy and recall.