Zero Defect Manufacturing (ZDM) strategies focus on promptly and precisely detecting defects to reduce material and energy consumption, and avoid product failure while the product is in use. Integrating a computer vision defect detection system is an important approach to improving the quality of inspection policies and the performance of the manufacturing process. Last year’s high attention has been paid to surface image defect detection based on deep learning algorithms. The deep learning methods on automated vision systems require large quantities of annotated data. Acquiring defects is a costly and time-consuming task in industrial contexts since defects only occur in small percentages, data is biased through the predomination of non-defective samples, and there is a lack of publicly available datasets to train the algorithms. This study aims to evaluate the performance of a deep learning algorithm based on the dataset employed and the training parameters selected.