As an important target for battlefield reconnaissance and maritime trade, the accurate detection of ships is of great significance for national security and economic development. In optical remote sensing images, visible images have clear color and texture characteristics, but are susceptible to cloudy weather or other adverse climatic conditions, and infrared images can reduce these effects to a certain extent. In this paper, the YOLOv8 is extended to a dual-stream network and the dual-modal fusion method of mid-term fusion is used, and the self-attention of the Transformer encoder is used to perform feature fusion between modalities in order to solve the problem that the traditional convolutional neural network is difficult to capture the long-distance dependence and global relationship between different modalities. In order to adaptively combine cross-modal features to enhance semantic information, a selection fusion module based on attention mechanism is used to capture rich semantic features, low semantic features are filtered by adaptively assigning weights to modalities, and rich semantic features are selected by ranking the importance of intramodal features. The effectiveness of the algorithm is verified by applying the algorithm to the ship target detection task of infrared and visible light images of multispectral remote sensing image datasets.