To address the challenges of low accuracy, high computational cost, and slow inference speed in current road pothole detection algorithms, this paper introduces a lightweight instance segmentation model for road potholes, called HLM-YOLO. The first improvement involves replacing the backbone of the YOLOv8 model with HGNetv2, a more efficient architecture that reduces both the computational load and the number of parameters. Next, the model integrates the MLCA (Mixed Local Channel Attention) mechanism, which combines channel and spatial information along with global and local context, to enhance the model’s ability to extract features of potholes at different scales, without sacrificing efficiency. Lastly, a new detection head, LSCD (Lightweight Shared Convolutional Detection Head), is introduced to replace the original one, effectively merging multi-level features while reducing the number of parameters. Experimental results show that HLM-YOLO achieves an average precision (mAP0.5) of 93%, which is 1.3% higher than the original YOLOv8n model. Additionally, the computational cost, parameter count, and model size are reduced by 25%, 45%, and 42%, respectively, while the detection speed increases from 64 FPS to 91 FPS, making the model suitable for lightweight and real-time road pothole detection.