Polarization-Sensitive Multimodal Network Based on Cross Attention for Object Detection

Multimodal information fusion can compensate for the perception limitation of unimodal systems by integrating multiple sensory data. Polarization can help distinguish objects with different surface materials, thus enhancing the reliability and robustness of object detection by fusing polarization features with visible light images. Due to hardware limitations, few high-quality polarization datasets are publicly available, especially for object detection, this paper construct a high-quality polarization-RGB multimodal object detection dataset PFDB based on the law of polarization features. To utilize the explicit expression of the polarization features, the coordinates of the luminous region of the polarization image are introduced as one input of the network. To capture complementary information and potential interactions between the different modalities, a three-modal multi-linear network PSMNet was proposed, which can deeply learn the feature of RGB, DoP and coordinates of the luminous region in DoP. PSMNet introduces a multimodal fusion transformer module in the network structure to globally interact the multi-scale fusion information. Comparison and ablation experiments demonstrate that the proposed method can effectively utilize polarization features to enhance object detection.