Wheat spike detection and counting is a critical step to ensure wheat yield estimation. In order to alleviate the problem of low accuracy of wheat spike detection in complex field backgrounds, this paper proposes a new model named WHDY based on YOLOv8. Firstly, this paper constructs a new wheat dataset, which contains 1200 wheat pictures and 7120 labels of wheat spike. Second, the WHDY model combines the CBAM attention mechanism, the WIoU loss function and the DCN variability convolution, which enhances the model’s detection precision for wheat spikes, optimizes the loss weights of small targets, reduces the interference of irrelevant factors in the complex field background, and improves the model’s detection accuracy. Finally, WHDY is compared with other models in a comparison experiment to verify the superiority of the model. The WHDY model achieves a mean accuracy of 96.1% with an inference time of 3.6ms. It effectively identifies and counts wheat spikes against complex backgrounds, offering a technical solution for wheat yield estimation.