Multispectral object detection aims at detecting targets with multiple spectral modalities, i.e. RGB and infrared images, to improve its reliability and robustness in harsh environments. Existing methods overlook the decoupled modeling of cross-modal fusion, making it difficult to select task-specific information from the multiple modals, resulting in false negatives. To this end, we propose a two-stage modal feature enhancement method that decouples cross-modal fusion into two parts, i.e. intra-modal and inter-modal feature interaction, to enhance modal features for multispectral object detection. First, for intra-modal feature interaction, we design a low-rank feature enhancement module that enhances the difference between target and background by suppressing irrelevant information in low-rank space. Second, for inter-modal feature interaction, we introduce a query-guided cross-modal feature enhancement module, which leverages modal-specific queries to retrieve modality-relevant information from intermediate features. Experimental results demonstrate that our method outperforms existing methods on benchmark datasets.