Surface defect detection of photodiodes is crucial for ensuring the stability and reliability of device performance. It effectively prevents potential failures, improves product quality, and enhances production efficiency. This study addresses the surface defect detection of photodiodes by constructing a multi-class defect dataset and proposing a hardware-accelerated detection method. First, a 16-bit fixed-point quantization and batch normalization (BN) layer fusion strategy were employed to optimize and train the YOLOv4-tiny model, improving its computational efficiency. Next, based on the ZYNQ7020 hardware platform, hardware architectures for the convolution, upsampling, and pooling modules were designed. The efficient hardware acceleration of the neural network was achieved using multi-channel parallel convolution operations and multi-path parallel data transmission techniques. Experimental results demonstrate that the proposed design achieved excellent detection performance on a test set of 100 photodiodes surface defects, with an average precision of 90.4%. The system can perform photodiodes surface defect detection on FPGA with a frame rate of 3.5 frames per second and a power consumption of only 3.08 W.