Accurate detection of sperms and impurities is a highly challenging task, primarily due to the small size of the targets, uncertain morphologies, and the presence of numerous impurities with varying sizes and shapes. Currently, the detection of sperms and impurities still relies heavily on traditional image processing and detection techniques, which have limited performance and often require manual intervention during the detection process, thus increasing time costs and injecting subjective bias into the analysis. Drawing inspiration from the numerous successful applications of deep learning techniques in various object detection challenges, we propose here an improved Yolo-v7 network model for sperm and impurity detection. This model employs Switchable Atrous Convolution (SAC) and the Wise-IoU loss function. Experimental results demonstrate that the highest AP50 for sperm and impurity detection are 95.1% and 62.4%, respectively. This significantly improves the detection accuracy of sperms and impurities, surpassing competitors and establishing state-of-the-art results on this issue.