In multi-modal object detection tasks targeting visible light and infrared light perspectives, existing methods are either limited by their bulky architecture and lack of cross-modal information interaction at the semantic level, or they integrate all information at the feature level without filtering, ignoring the interference of redundant information. Based on these issues, we propose a cross-modal object detection algorithm based on a sparse self-attention mechanism. A twin backbone is utilised for the simultaneous encoding of feature maps of different modalities. This is accompanied by the introduction of a sparse self-attention module and an adaptive proportion modulation module. The refinement and filtration of cross-modal information at the semantic level is thereby achieved, with the attention weights being adaptively adjusted based on the distribution of multi-modal feature domains. This process culminates in high-precision multi-modal object recognition. A series of experiments, comprising both qualitative and quantitative analysis, were conducted on multiple benchmark datasets. These experiments were designed to assess the performance of our network, a deep learning model, and the results obtained demonstrated its superiority. On the M3FD dataset, our model achieved an mAP of 90.4, while on the FLIR dataset, it attained an mAP of 87.0.