In construction sites, the high fatality rate from falls due to not wearing safety belts underscores the need for real-time detection of safety belt usage at heights. Current detection methods for high-altitude work face challenges such as low accuracy, high computational demands, and large model sizes, which hinder real-time application. To address these issues, this paper introduces a lightweight model, FMS-YOLO, which utilizes a fused mixed local channel attention (MLCA) mechanism. Key strategies include integrating the lightweight FasterBlock module into all C3 (Cross Stage Partial Network) modules to accelerate inference and reduce parameter count; incorporating MLCA into the Neck for enhanced feature extraction and improved detection accuracy; and replacing the CIoU loss function with Shape-IoU, which focuses on the shape and aspect ratio of bounding boxes to enhance model robustness. Experimental results show that the proposed model achieves an mAP@0.5 of 96%, with only 5.4M parameters and the size of the weight file is only 11.3 MB. Compared to the original model, the proposed model achieves a 3% improvement in mAP@0.5 and a reduction of 1.2M parameters. This model meets the real-time detection requirements for monitoring the safety belt usage of workers operating at heights in construction site surveillance videos.