There is a high demand for detecting defects on the surface of objects. In industry, inspecting mass-produced items is costly and labor-intensive. This paper introduces an automatic defect detection method. Specifically, the method detects defects on multi-colored objects. While gradient-based edge detection can detect defects on grayscale images that have a uniform non-defect region, the edge detection approach does not work for multi-colored objects because gradient changes exist in non-defect regions of the object. The method proposed here detects defects on multi-color surfaces by generating an ideal defect-free surface. An analysis of eight 200*200 samples shows that on average, the defect detected by our algorithm has a 0.9193 Dice Similarity Coefficient (DSC), 0.8634 Intersection over Union (IoU), and 3.3995 Hausdorff Distance (HD) compared to hand-labeled defect. This algorithm performs better than common edge detection algorithms when detecting defects on non-uniform color surfaces.