In the study of rock microstructure, Scanning Electron Microscopy (SEM) images are commonly used for identification. However, the grayscale differences between pores and different rocks in the image are different, and the shape and size distribution of rock pores are not uniform. The pixel value of small-sized pores is relatively small, which often increases the difficulty of recognition. To address the complexity and low accuracy of rock microstructure identification, this paper proposes a small object detection algorithm based on artificial intelligence deep learning YOLOv8. The algorithm significantly improves accuracy of the model in detecting small pores and achieves good coverage for targets of all sizes. The network design incorporates a new feature extraction method to preserve shallow feature information and introduces a CBAM attention mechanism to improve network performance. The backbone network utilizes Pconv convolution and introduces the P_C2f module to reduce parameters and increase computational speed. The Wise-IoU loss function is employed instead of the CIoU loss function used in YOLOv8. The experiment shows that the YOLOv8 network construction model proposed in this paper outperform traditional methods in rock microstructure detection, with a 30% increase in detection accuracy and a fourfold increase in efficiency. This algorithm provides a new approach to rock microstructure identification and detection.