The assurance of factory personnel safety has become increasingly emphasized in the contemporary industrial system. Modern industrial construction necessitates a comprehensive personnel safety assurance system, and the ability to achieve real-time monitoring of factory personnel behavior is a prerequisite for establishing an effective safety system. In this paper, we present an improved YOLOv5s model to handle complex occlusions and the broad size range of detection targets commonly found in real-world production settings. By incorporating Deformable Convolutional v4 (DCNv4) into the C3 module, a new C3-DCNv4 module is created to improve the model’s adaptability to complex environments. Additionally, the Bidirectional Feature Pyramid Network (BiFPN) structure is integrated into the neck of the network to enhance the model’s detection capabilities for targets of varying scales. The experiment shows that on the factory safety dataset, the improved model’s mean Average Precision (mAP) for detecting unsafe behaviors is 85.2%, representing a 3.8% increase over the benchmark model. Additionally, the computational load decreases by 0.6 GFLOPS. The enhanced model delivers outstanding detection results with fewer computational parameters, making it an efficient choice for real-time industrial safety monitoring.