A lightweight obstacle detection model based on an improved YOLOv5s detection network is proposed to address the shortcomings of traditional train track obstacle detection methods, such as poor real-time performance and low detection accuracy for small targets. On the basis of introducing linear bottleneck and reverse residual structure, the backbone network, together with SE, forms a new network module MobileNetV3, which makes the model lighter without reducing the accuracy rate. Starting from the feature fusion of neck, a weighted bidirectional feature pyramid BiFPN is introduced to enhance the underlying information of feature maps, enabling information fusion of feature maps at different scales, thereby enhancing feature information. Adding CA spatial attention mechanism at the end of the model feature extraction network reduces the loss of important position information during the training process. The experimental results show that the improved model improves the detection accuracy by 1.2 percentage points compared to the original YOLOv5s algorithm on a self-made diversified rail transit dataset, while reducing the model size by 4.1 MB and increasing the detection speed by 7 FPS. It has certain feasibility and superiority in detecting track obstacles.