Orbital-Side Foreign Object Intrusion Detection Network Based on Transformer Multimodal Fusion

With the continuous expansion of China’s railway network, accurate detection of foreign object intrusions along railway perimeters has become a critical research challenge. To address the limitations of single-modal vision-based detection systems—particularly their poor robustness and inadequate performance in complex environments—we propose an innovative multi-modal fusion detection algorithm. Our approach simultaneously utilizes visual and auditory data, employing a transformer-based architecture to effectively fuse these modalities. The algorithm preserves modality-invariant features while enhancing modality-specific characteristics, with visual inputs serving as the primary modality and audio data acting as a supplementary modality to reinforce detection capability. This framework enables reliable identification of track intrusions by foreign objects. Extensive experiments demonstrate that our solution achieves outstanding performance on proprietary datasets, meeting both real-time processing requirements and high accuracy standards. The results indicate significant potential for practical engineering applications in railway security systems.