Robust Environmental Perception of Multi-sensor Data Fusion

In this chapter, the multi-sensor data fusion based neural networks are adopted to environmental perception fault tolerance algorithms. Multi-sensor fusion can solve the problem of perception reliability when some sensors fail by using data redundancy and improve the robustness of the perception algorithms. In Sect. 2.1, a 2D-3D pose estimation network based on keypoints is proposed, which can be applied to calibrate the camera and LiDAR in real-time. In Sect. 2.2, a real-time data fusion network with fault diagnosis and fault tolerance mechanisms is designed. By leveraging temporal and spatial correlations between sensor data, this network utilizes sensor redundancy to diagnose local and global confidence of sensor data in real-time, eliminating faulty data and ensuring accuracy and reliability of data fusion. In Sect. 2.3, a novel multi-phase fusion network for robust 3D semantic segmentation is proposed. Three factors that restrict the performance of fusion-based 3D semantic segmentation methods are summarized: inefficient feature fusion mechanism, inability to effectively express features, and lacking of dense labels, and corresponding solutions are put forward. The usefulness and advantage of the proposed algorithms are demonstrated via experiments on common datasets and real scenes.