The state of track components directly determines the safety of train operations. Existing intelligent detection algorithms based on RGB images can accomplish the task of track defect detection in most scenarios. Due to the limited information embedded in the RGB image, it is susceptible to leakage and misdetection in scenes with high similarity between the background and foreground. Depth images can provide spatial location and distance information of objects to make up for the lack of RGB. This paper proposes a multimodal fusion algorithm for track defect detection named MSDEYOLO. It utilizes a parallel architecture to co-process RGB and depth data for defect detection. Firstly, we propose the C2f_EMBiFocus module to enhance the performance of the backbone in feature extraction. Secondly, we design an additional small object detection head to improve the algorithm’s sensitivity to small objects. Finally, we employ SIoU Loss to accelerate the convergence speed of the algorithm and improve its regression accuracy. Experiments on the track multimodal dataset show that the mAP@0.5 of the MSDEYOLO algorithm reaches 93.1%, and the mAP@0.5–0.95 reaches 71.6%. Compared to other state-of-the-art multimodal and unimodal object detection algorithms, MSDEYOLO demonstrates the highest accuracy and robustness in complex railway scenarios.