The issue of railway intrusion has always been a major concern for railway safety. Solving this problem not only ensures safety, but also effectively avoids situations where trains slow down or even brake urgently due to encountering obstacles, thereby reducing the additional carbon emissions caused by braking. To further improve the detection accuracy of pedestrians in railway intrusion problems, this study proposes a novel model based on the YOLOv8 model for improvement. This study introduces attention mechanism, new convolution method and upsampling method based on the original model, and adds a small object detection layer to specialize the model’s perception of small objects, improve the accuracy of small object detection, and solve the problem of low recognition accuracy caused by pedestrian objects being too small due to various reasons. At the same time, it also achieved lightweighting of the model, making it easier to configure on the detection site. Through improvement, This study has significantly improved the performance compared to the original model, achieving 82.1% precision and 76.8% mAP50, while reducing the number of parameters by 22%. This fully demonstrates the superiority of the improved model performance.