In response to the current issues of high resource consumption and low identification accuracy in highway pavement diseases detection, an improved YOLOv5-based (You Only Look Once Version 5) algorithm combining MSDA (Multi-Scale Dilated Attention) and LSKA (Large Separable Kernel Attention) attention mechanisms within the KAN (Kolmogorov–Arnold Networks) is proposed. On the basis of improving the YOLOv5 image recognition algorithm, this paper also adds a genetic algorithm to the original algorithm structure to optimize the parameters involved in the model. In addition, image enhancement methods such as HSV and Mosaic are used to expand the experimental sample materials, which helps to enhance the richness of pavement diseases features. The results indicate that the improved YOLOv5 image recognition algorithm we proposed increases the identification accuracy for various types of highway pavement diseases such as cracks, repairs, and potholes, with an overall improvement in precision and recall compared to the original network. This improved algorithm achieves good recognition effects for the automated detection of actual highway pavement diseases and well meets practical needs.