Enhancing Lidar and Radar Fusion for Vehicle Detection in Adverse Weather via Cross-Modality Semantic Consistency

The fusion of multiple sensors such as Lidar and Radar provides richer information and thus improves the perception ability for autonomous driving even in adverse weather. However, Lidar paints 3D geometric point cloud of the scene while Radar provides 2D scan images of Radar cross-section (RCS) intensity, which brings challenges for efficient multi-modality fusion. To this end, we propose an enhanced Lidar-Radar fusion framework, Cross-Modality Semantic Consistency Networks (CMSCNet), to mitigate the modal gap and improve detection performance. Specifically, we implement the semantic consistency loss by leveraging the concept of Knowledge Distillation for cross-modality feature learning. Moreover, we investigate effective point cloud processing strategies of adaptive ground removal and Laser-ID slicing when transforming Lidar point cloud to the BEV representation that closely resembles the Radar image. We evaluate the proposed method on Oxford Radar RobotCar dataset of simulated fog point cloud and RADIATE dataset of real adverse weather data. Extensive experiments validate the advantages of CMSCNet for Lidar and Radar fusion, especially on real adverse data.