Quality control in semiconductor manufacturing is critical for ensuring the reliability of electronic devices. Among the various challenges faced, the detection of surface defects on processor sockets remains a significant hurdle due to the limitations of manual inspection methods. This study introduces a machine learning-based approach to automate and enhance the accuracy of surface defect detection in processor sockets. We detail the development and implementation of a convolutional neural network (CNN) model, which leverages transfer learning techniques to overcome the constraints of limited and imbalanced data. Our methodology includes a comprehensive data acquisition process using high-definition digital microscopy, followed by data augmentation strategies to ensure robust model training. The results demonstrate that our model achieves a testing accuracy of 95%, significantly reducing the likelihood of defective processors entering the production line. Furthermore, we explore the optimization of our model using the OpenVINO toolkit, achieving a notable improvement in execution speed without compromising accuracy. The implications of this research extend beyond the semiconductor industry, offering potential applications in various mechanical engineering domains.