Remote sensing images are commonly characterized by complicated backgrounds and dramatic scale variations, presenting substantial challenges for object detection. While mainstream object detection networks perform well in detecting generic objects, they struggle with remote sensing object detection due to three primary challenges: insufficient semantic feature extraction, feature fusion confusion, and potential cross-layer information loss. To alleviate these problems, we develop a semantics-driven multiscale selective fusion network (SMSF-Net) for object detection in remote sensing images. First, we introduce a Global Semantic Information Extraction Module (GSIEM) to suppress the complex background noises by capturing multiple receptive fields from a deep feature map. Next, we design a Multi-Dimension Adaptive Integration Module (MDAIM) to address the issue of feature fusion confusion through feature alignment and adaptive information fusion. Furthermore, a Multiscale Selective Feature Fusion Network (MSFFN) is proposed by integrating the MDAIM with Gather-and-Distribute Mechanism to prevent cross-layer feature loss. Finally, we incorporate four detection heads to enhance the network’s capability for multiscale object detection. Extensive experiments on the DIOR and SIMD datasets confirm that the proposed SMSF-Net effectively excels in detecting remote sensing images with complex backgrounds and multiscale targets.