On construction sites, wearing a safety helmet is a crucial and effective measure to protect workers from unforeseen injuries. In order to ensure timely reminders for workers to wear safety helmets, it is necessary to automate the detection process of safety helmet-wearing. However, Limited to the camera distance and environment, detecting relatively small targets can be challenging, this study constructs a safety helmet-wearing detection algorithm based on MobileViT and Centernet. Firstly, in the feature extraction stage, a more lightweight MobileViT algorithm is employed to replace the original Resnet50 algorithm. Additionally, the efficient channel attention (ECA-Net) module is introduced in the first two layers to weight feature information, thereby reducing the degree of target information loss. Secondly, the introduction of the ASFF-FPN structure strengthens the network’s cross-scale feature fusion capability, enabling it to effectively detect small targets. Lastly, a lightweight CARAFE upsampling operator is employed for upsampling operations to reduce the model’s parameters. The final model achieves an average precision of 86.37% in the task of safety helmet detection, with an increase of 6.24% compared with that of the original Centernet. On the GeForce RTX 3060 Ti graphics card, there is a 17.8 FPS improvement.