Multispectral object detection has drawn much attention as color-thermal modalities can provide complementary information. However, it is significantly difficult to produce a reliable prediction because of the modality imbalance problem. To deal with this problem, this paper introduces a novel Attention-Guided Uncertainty-Aware (AG-UA) method. Specifically, we first propose an Attention-guided Feature Recalibration (CFR) module to adaptively recalibrate features by fully leveraging complementary information to deal with the feature modality imbalance. Secondly, we propose an Uncertainty-Aware Fusion (UAF) module to adaptively fuse the Region of Interest with estimated uncertain scores to cope with the illumination modality imbalance. Extensive experimental results show the effectiveness and state-of-the-art performance of our proposed approaches.