Detection of safety harnesses is essential to ensure the safety of workers in high-altitude operations such as elevator maintenance and construction hoisting activities. However, existing detection methods mainly focus on detecting the presence of the safety harness, without assessing whether it is worn correctly. To solve this problem, the semantic segmentation-based intelligent safety harness detection method is proposed for elevator maintenance operations, which utilizes Residual Concatenated Convolutional Block Attention Module (RC-CBAM) and Temporal Fusion Attention Module (TFAM) to enhance the sensitivity of the U-net network. The extracted features from TFAM are aggregated in the channel dimension to obtain segmentation results that match the resolution of the input image. Convolutional Neural Networks (CNN) is used to evaluate the correctness of the safety harness wearing. Experiment results demonstrate that the attention mechanism-based U-Net outperforms the U-Net. The proposed method can accurately detect the incorrect wearing of the safety harness.